May I know whether Matillion ETL supports Snowflake Data Pipe - Snowpipe?
I tried reading Matillion documentations but couldn’t find it anywhere.
Thanks
Snowpipe is an object created in Snowflake. Once created, it will continously ingest data once there is data in the stage on which it is created. So there is no need of scheduling it or running it everytime.
You can run the DDL of the Snowpipe onetime from Matillion for Snowflake using the SQL script component.
Hi @winston.l ,
Snowpipe can be used for a few different loading styles but it's commonly used when there is a need to take streaming data and ingest it into Snowflake in near real-time. This is what @pratikdatta1 is referring to. The other thing about Snowpipe is that it's typically a set it and forget type thing. Once it works no scheduling and so on is needed depending on your situation.
Matillion can do something similar using CDC (change data capture) but there is more moving pieces and overhead there from a cost perspective. With that said, there are use cases where this is simply the best solution due to where or how the source data is stored.
Although there is overlap between Snowpipe and CDC replication, they really are different and there isn't much that Matillion could offer to help Snowpipe do it's job better.
I hope this helps!
Thank you @pratikdatta1 which I thought so too that Matillion components may not need to show this