Database Query Component - Going into a hung mode when extracting moderate volumes of data

Matillion ETL
1

Hi all,

​

We are using Database query component to extract data from on-premise database using a IPSec VPN tunnel to AWS and load data into Snowflake.

​

Recently, the component is going into a hung mode while fetching volumes more than 30k records. The component hungs up while staging the records into S3 staging Area and does not proceed further. Has anyone else encountered this issue before ?

​

​Network connectivity seems okay between AWS and On-Premise server.

​

Cheers,

Anu​

2

Hi Anu,

 

we have lots of replication tasks with the Dabase Query Component. These jobs transfer Millions of rows every day (the largest job currently stages ~10M rows per day). Our source systems are connected via several heterogeneous VPN tunnels. We haven't seen any issues lately.

 

However, we haven't updated our instance for some time and are still on 1.63. Did the issue appear on your end with a more recent version?

 

I think the best you can do is to open a support request and send over the log files.

 

Michael

3

Hi @Michael​ , Thanks for the response. We too did not experience any issues with Matillion fetching large volumes of data from on-premise systems previously.

However, we realised it is not Matillion's problem. We tried to replicate the process of data extraction using SQL Tools on EC2 Instance to trying to understand if it is Matillion/Network issue and had the same problem. It seems like there is an AWS parameter setting (lifebytes) on the VPN Connection, which is limiting the amount of data transferred at single go (typically set at 4.6 mb). Once this setting is disabled, we are now able to fetch the higher volumes.