How to handle schema change in Incremental Load model in Snowflake

Matillion ETL
1

Hi everyone,

 

What is the best way to update a target table whenever there is an addition of columns in respective source tables, without doing a full load.

 

We have monthly column additions to some source tables and this breaks the incremental load job in Matillion as it cannot recognise the schema change in source tables, probably because it does not match with the schema of target tables.

 

Has anyone face this issue before and if yes, then how did you solve the problem? Currently, I initiate a full refresh of a table on source schema change, but I prefer not to follow this method as we have some tables having nearly 50 million records, a full refresh of them is definitely not an ideal solution.

2

@NeelamMacwan​ Have you looked in the JDBC Incremental Load wizard component. It has capabilities to handle schema drift.

https://documentation.matillion.com/docs/2955330#enable-schema-drift

3

Thank you so much @ramalingam​

I'll check this out.