Dear Matillion ETL Support Team,
In our current organization we are currently working on projects where we extensively use shared jobs as generic modules within our Matillion ETL workflows. We’ve encountered an issue when trying to pass variables from a shared job component to subsequent components in the job sequence.
Scenario:
- We have a shared job that includes a Python component.
- Inside this Python component, we execute a SQL script and assign the query outcome to a context variable using 'context.updateVariable(variable_name, outcome)'
- The problem arises when the shared job completes and the flow moves to the next job or component in sequence (e.g., another Python script). When we try to access the context variable using context.getVariable(variable_name) or by referencing variable_name directly, it is not recognized.
Our Understanding:
- We are aware that in Shared Jobs integration, input variables can be accessed in other components using the Export section and by setting the target variable with another job variable. However, our use case requires that we dynamically update context variables within the shared job and then access them in subsequent components outside of the shared job.
- It seems that the context of the shared job is isolated, meaning any variables updated within the shared job's context are not accessible once the shared job execution is complete and control is passed to the next component.
Our Concern:
- We are using shared jobs as modular, reusable components and building custom solutions on top of them. We prefer not to create additional shared jobs for every subsequent task simply to maintain the context. This approach would reduce the reusability and scalability of our solutions.
Could you please provide guidance on how we can persist context variables updated in a shared job so they can be accessed by components outside of that shared job in the same workflow? Any advice or best practices you can share to address this issue would be greatly appreciated.
Thank you for your assistance.
Best regards,