How to pass variable value back to calling job

Matillion ETL
1

I have an orchestration job that transfers files from a windows fileshare to Azure blob storage. which fileshare and which files (regex filter) is passed in via variables.

The job is called from various other jobs (each project related and independant of eachother).

The transfer job will on each run generate a unique foldername and then store the transferred files in that subfolder on Azure.

 

I need a way to pass that unique foldername back to the calling job so it can go and fetch the files that were transferred for it and process them into Snowflake.

 

Is there a way to pass variables back up to the caller?

I have tried with public and shared, but that only seems to work within the job itself.

 

An environment variable is not reliable because jobs may run in parallel and the env var would get overwritten by a different run of the same job.

 

I would rather not use a db table as a communication channel, as it will get filled with passed run information over time.

2

I just accidentally discovered it myself...

 

There is an 'export' tab in the properties window of the 'Run Orchestration' component.

In that tab, you can decide which variables from the child job (callee) are sent back to the parent job (caller) and in which variable of the parent they must be stored.

 

I've been looking for this for many hours in the past days, and it turns out to be very simple.

 

Special shoutout to Prabhjot Kaur Thind for documenting this in the article 'Variable Communication in Matillion Jobs' on her linked-in page.

3

pretty please to matillion documentation writers: please mention the export functionality in your section about variables.