I'm looking for an efficient way to consolidate error messages from child jobs within my Matillion orchestration. I want a single email alert at the orchestration level that will trigger if any child job fails. Can you suggest a streamlined approach?

Matillion ETL
1

I have prepared a Batch Pipeline Process. I wanted to create an email alert whenever a child job fails its Email Alert get rolled up to the orchestration level. I have seen documentation of using Export Variables of Component to be rolled up to the orchestration level Please find below link:

 

https://www.matillion.com/blog/matillion-etl-error-handling-patterns

 

this is a very tedious task as to apply this functionality in every job. Another Document i have seen is about ETL Task History data to be stored in the table and use that to create email alert of batch failures.

 

that also is not sustainable. "Send Email" Component is ready to use.

 

Can you suggest how can i create a send email alert in a way that i attach "Send Email" Component in front of Orchestration (Parent) Job so that when its Transformation (Child) job fails, it gives the error message of its transformation job as well as its parent Job.

 

Find below in the Batch Pipeline Process Image.

 

or Can you suggest any other way to do Email Alert Failures?

 

069Tm000002ywHlIAI_0D5Tm000006eMuSKAU
069Tm000002ywHlIAI_0D5Tm000006eMuSKAU1801×682 142 KB

2

Hi,

You can attach you send email component in the failure node of the run orchestration. if your Orchestration fails it will run the send_email component.

3

This is one of the critical issue which we had while implementing a batch process in Matillion. When you have Orchestration of Orchestrations, capturing the error at the lowest level or the top most level is always critical.

 

One of the solutions we had to overcome this issue is , "Using a Template Orchestration Job". Ideally , you could have created a Matillion Shared job which will execute your orchestration and throw out an error , when you encounter a failure. and use it for running each of your orchestrations. However, Matillion doesnt allow to run Orchestration jobs from Shared jobs. So , we had to create a template job , which does the below steps .

 

  1. Initiate the Orchestration job/Job to executed. This can be done in one transaction.
  2. Run the Orchestration Job
  3. Log the results/outcome.
  4. Report Error if any
  5. End Successful/Failure.

 

This template job is used to be called by over -arching Orchestration jobs by passing the executable Orchestration job as parameter.

 

Hope this info helps.

 

 

4

Yes, i have tried that. So what i have understand using Matillion is that if a child job fails, its orchestration job also fails but the Child job Error message where failure has occurred, its error message does not roll up to its orchestration job. In your scenario, Send Email component will work but its subject part will be blank, 

 

After reading Matillion best practice document, way around is that you use Export Variables of Components (linked shared in above message) to be rolled up to the orchestration level but that is a very tedious task to be done with every job.

 

that is why looking for more affectioned and streamlined approach or what i am missing in this whole process?