What are some techniques to Trigger a transformation job once a table is updated in Snowflake via a REST API?

Matillion ETL
1

Hello All,

 

Basically, I want to run a transformation job in realtime once a record has been inserted into the SF table and I want to avoid having something scheduled to run every X minutes.

 

any ideas?

 

TIA!!

2

Hi @Adwait_K​ ,

I don't think this is possible as Snowflake lacks a "Trigger" feature. Even with their own approach to ETL pipelines ("tasks") the use polling on the tables instead of an event-based system.

Let me know if your reasearch ends with a different result, but I think the options for event-based data processing with Snowflake and Matillion are limited.

3

Hi @Michael​ . Yeah, that's what I figured as well. Thanks!

4

You can use Snowflake Streams to capture the inserts on the Snowflake Table. Run your pipeline when the Stream has data available in it. However, you still need to schedule your orchestration to run every x minutes. There is no way currently to trigger events based on actions on a table in Snowflake.

5

Thanks @AnudeepK​