Matillion server filling up quickly

Matillion ETL
1

Hello All,

 

We have a 40G matillion Ec2 running and 33G are bring taking up by the directory:

/var/lib/pgsql/9.6/data/base/

 

All the files look like they are logs. Looking inside the files, they are keeping step by step records of everything going on in the server - CDC runs, SQS runs, orchestration runs, etc.

 

Funny thing is the user & group creating these files is "postgres". Is this something internal to Matillion?

 

Thanks!!

 

2

Hi

 

Have you found any further information on this? Our /var/lib/pgsql/9.6/data/base is up to 30GB now....

3

Hi @Adwait_K​ & @zelman.yuen​

The following article has advice on cleaning up disk space by adjusting task-retention settings: https://support.matillion.com/s/article/How-To-Reclaim-Free-Disk-Space-from-Postgres-Database

Additionally, note that the article instructions mention postgres13 as our newer images ship with that. It looks like you are on an older version of Postgres. One option could be to plan for a migration to a new VM, which will get you to the latest version.

Again please ensure that you are on postgres13 and new VM which will not have historic data which means more disk space.

Hope this helps,

Chika

4

Thanks @ChikaMatillionCommunityMgr​ . This is pretty much the same info we got from Support when we opened a ticket. It fixed the issue for us.

@zelman.yuen​ Couple of other things you could potentially look at:

How To: Change Task History Retention (matillion.com)

How to configure Catalina log rotation | Matillion ETL Docs

Hope this helps.

Thanks.