Is there an easier way to view Server logs?

Matillion ETL
1

Hello All,

 

SQS Listener logging information is only logged in the Server catalina logs. Currently, the only option is to go to Admin->Download Server Logs to view the catalina logs. But this downloads a LOT of information that is not relevant to me.

Do others have an easier way of dealing with massive server logs and using them to troubleshoot issues?

 

Thanks!!

 

2

Hi @Adwait_K​, I may or may not be able to give you some information that will help. I will start by saying we don't currently use SQS so take my thoughts with a grain of salt. Is your instance large enough to enable the Enterprise Mode of Matillion? If so, does the SQS logging information end up in the Audit Log? If so, you could use the API to extract that from Matillion and store it in a table. Just another place to look.

If this is not the case, I would definitely post a request to expose the SQS Listener logging info via the API in the Ideas Portal. It seems like a valuable thing to have because as you stated dumping the Catalina logs has it's own challenges since they can be so large and have more information than needed. I hope this helps in some fashion. If you think we can help more please feel free to reply.

3

Hi @Adwait_K​ , if the mere size of the catalina log file ist your main problem, I can probably help. Some versions of Matillion have an issue with the inbuilt logrotate setup and log files grow and grow forever. On our instace, the log was more than 1.5GB in size which was impossible to handle. I could post a workaround to keep the log file size more feasible.

If your main issue is that you don't want to look at the log in a plain text file structure, I can't really help you. You could try to write a small parser, maybe in Python, to load the log file structure to a relational database for better log file analytics. Maybe a quite simple regex can do the job. I haven't tried myself but if you come up with a solution, please share it here. It might be useful for others, including me ;-)

4

Atleast in AWS the Matillion instances are 'sort-of' running the 'awslogs', which can log the entries from catalina.out to AWS CloudWatch.

I believe the agent was initially sort of stuck and awslogs needed to be reinstalled and the agent removed. Can't really remember the path I went to get it work, but google helped me. :)