I am using Matillion installed on GCP to connect to AWS RDS Postgress and the connection duration to RDS Postgres is a max of 5 minutes. So I want to chuck the data that I fetch from RDS data and append the Snowflake database in GCP.
Hey there!
So first off, I gotta ask what is causing that max 5 minute TCP connection and whether that is in your control at all? The component advanced properties has options for setting things like "keepalives" and whatnot so maybe all of this pain can be avoided.... but if not read on!
I would approach this by finding a column that had only a few values (or a numeric than could be split into non-overlapping ranges) to break up logically into X parts. An "iterator" on top of the data staging component could then pass those values into the component, which could filter on that value in the column. You'd end up with multiple datasets on GCP, one for each set of values (or range of number) but you can union those together into a single view (Create View) and treat that like the original dataset in bigquery.
Whether you have a suitable column is gonna make or break it I think!
Let me know how you get on...
@david_langton Thanks for the suggestion, I will explore the options mentioned by you.
Just to add, I meant "Snowflake" at the end, given you're on GCP I said BigQuery without thinking! But it would be the same deal.