Incremental Load Component

Matillion ETL
1

Hello All,

 

Could someone please explain how the Incremental Load component, specifically the one for Salesforce works behind the scenes?

 

My expectation was that the component is intelligent enough to keep track of "Last Run Time" and then issue a query to the Salesforce object in question to pull in records that were modified since this timestamp.

 

But my observation is that regardless of the last run time, the component stages ALL the records in S3/Snowflake and then does a MERGE with the records already existing in the Snowflake table. This is highly inefficient obviously since it stages records that haven't even changed. What would be most efficient is to issue a query, for example, like this - select * from Account where lastModifiedDate > ${LastRunTime}

 

Anyone have any insights into how exactly this component works?

 

Thanks!!

2

Just a friendly bump to get more visibility.

3

The best way for solving this to maintain the high watermark timestamp in a table or file somewhere accessible to Matillion. Using Query result to Scalar component, you can read the last high watermark into a variable and then use it in your Incremental load component query to fetch from the source.