By Matillion how we can achieve incremental load?

Matillion ETL
1

Hi all,

So I have this scenario in my hand and I need a workaround regarding this using matillion,

We have s3 bucket where in every 2 hours it receives new set of file, By using matillion how can we get the newly added file and append it in the table? By Matillion how we can achieve incremental load?

2

If its snowflake Why not use Snowpipe directly with external storage integration to S3?

 

Snowpipe always looks at incremental only .

3

As Ranzith already posted, I'd also suggest to look at Snowpipe if you are on Snowflake. That works realy good and very solid.

 

If your file or folder names on S3 include a timestamp, there is another workaround: Use a "Query to Scalar" or a Python component to generate a load pattern for the S3 component which allows you to load all files with a matching timestamp from the last x days or y hours. This is no "real" incremental load but helps to keep the number of files to load quite small.