Matillion column names changed and broke pipelines

Data Productivity Cloud
1

We have a custom connector pipeline whose Matillion columns changed without warning from prefix MTLN_CYOC_ to MTLN_CC.

 

This broke pipelines to Snowflake because the cols did not exist.

 

Did I miss an update on this change?

2

Hi @egeorge​ and thanks for posting.

I'll check in with the team about this. In the mean time, if you have any logs for the broken pipelines, could you pass them over as it may help. Feel free to DM them to me if you'd prefer.

Many thanks,

Claire

3

Thanks. Here are the logs from the failed pipeline when the Matillion system columns changed.

4

Hi @egeorge​ and thanks for your patience while I checked with the team.

There was a change made to the product to support an enhancement that appears to have caused the issue that you are seeing. I do have a potential fix for you though. Please try running the ALTER SQL below. Please note that you will need the correct permissions to run this and you will need to run it for each table. If you find any other MTLN_CYOC prefixed columns, alter them in the same way. We will be adding this to our documentation, so thank you for bringing this to our attention. Let me know how you get on.

Many thanks,

Claire

-- Rename Matillion Audit Columns - run both statements for each Table

alter table "<database>"."<schema>"."<tablename>" rename column "MTLN_CYOC_PARTITION" to "MTLN_CC_PARTITION";

alter table "<database>"."<schema>"."<tablename>" rename column "MTLN_CYOC_IDENTIFIER" to "MTLN_CC_IDENTIFIER";