Download All Catalina log files

Matillion ETL
1

The download log button in Matillion only downloads the current catalina.out file, however this file does not contain the time span that i'm looking for. How do I download all the catalina log files?

2

Hi Botn,

 

You'll need to SSH onto your Matillion Linux server in order to get the older catalina.out files. They are not accessible via the user interface.

 

All the files are in /var/log/tomcat8 and you should find:

 

  • One named catalina.out (this is the current one, which you get if you run Admin / Download Server Log
  • Zero or more files archived by logrotate, containing earlier logs, named e.g. catalina.out-20210531.gz

 

There used to be a bug where logrotate was not enabled, resulting in just one enormous catalina.out file which would eventually fill up the entire disk. The workaround is documented here, although I think it has been fixed now.

 

JB

3

Do you also know the bash command to export the files to local desktop?

 

4

Sure, yes. Normally it has to be an import (a "pull") rather than an export, because you can't generally push files onto your Desktop from somewhere else.

Also there's a complication, which is that the files are owned by the Linux user tomcat, which can't SSH in. For security reasons I would not recommend changing the visibility of files in /var/log/tomcat8. Instead, log in as root, copy the file to /tmp and then scp it from there.

So, on the Matillion instance from an SSH session as the centos user

sudo -i

cp /var/log/tomcat8/catalina.out-20210531.gz /tmp/

chmod 444 /tmp/catalina.out-20210531.gz

Then, on your laptop

scp -i YourPrivateKey.pem centos@your.matillion.server.com:/tmp/catalina.out-20210531.gz .

That will copy the selected archived catalina.out file onto your local machine. The copy in /tmp on the Matillion server will disappear eventually, but as best practice I'd say delete it once you have made a copy.

JB

5

Hi @Botn​,

You don't specify what cloud provider you are using but if you are on AWS, another approach could be to create an Orchestration that dumps the Catalina logs to an S3 bucket using a bash script.

This is obviously an extremely simple example but could be modified to include more logic and processing:

cd /var/log/tomcat8/

aws s3 cp catalina* s3://your_S3_bucket_URI/some_folder/