I would like to monitor the number of concurrent users in our Matillion instances, so my first thought was to add the number of concurrent users as a metric in CloudWatch. The first problem is that I don't know where to find the actual metric, I can see which users are connected in the Matillion GUI and how many of them, but I can't find that information in Matillion's API. Any help would be greatly appreciated. Or if there is a simpler solution on how to monitor the number of concurrent users and set up triggers and thresholds would be really helpful.
Hi @SS129958, as far as I know the functionality you are looking for doesn't exist in the current API. The only places I can think of that information existing would be on the Ec2 instance itself or in a table in the Matillion Database. I looked through the tables in the database and didn't see anything that led me to believe it's in the DB. It's considered bad practice to query the database directly anyway.
If it's on the Ec2 instance then it's likely going to be in a file related to Tomcat which by default isn't going to output to Cloudwatch. You would have to manually write something to do that. If you knew the file that had the info, you may be able to call a bash script in a Matillion job to output the contents somewhere (like a table, S3, Cloudwatch, etc.). As of right now, I don't know that there is an easy solution.
I would suggestion submitting the idea of including user metrics in the Matillion API for instance management purposes. You can do that in the ideas portal. I know those ideas are monitored by Matillion pretty closely. Sorry, I couldn't be of more help.
I haven't spent much time in Azure as we do almost everything in AWS but I will try to help. I don't think it's necessarily a bad thing to have the connection string in the Python code but it is considered bad practice for many reason to store (in your case) the Account and AccountKey. I personally wouldn't store the container and blob within the Python code either. You should store the sensitive info in something like the Azure Key Vault or in the Matillion password store. The idea would be to retrieve the sensitive items from the Key Vault or Password store prior to calling your Python component and setting those as job variables. This effectively means you are not storing your sensitive credential type info with your code and you are on the fly grabbing that info as needed from a secure location and passing them into your Python script via a variable. As soon as the job finishes the credential info in the variables is gone.
If you use the Matillion password store, then you can retrieve the password via the Matillion API in a step before your Python Script. I'm afraid I can't be of much help if you are trying to retrieve the values from an Azure Key Vault as this is a situation where AWS is very different from Azure and I don't have much knowledge. This is task is extremely simple in AWS but I don't like to make assumptions about other cloud providers.
@UmeshS I'd be keen to hear how you got on? Finding little to no docs/blogs about leveraging azure blob storage via python on Matillion. All docs relate to AWS!
Accessing Azure Blob Storage in Python doesn't have anything to do with Matillion. If you can write a Python script or program that successfully accesses and works with Blob storage outside of Matillion then you can grab that same code and drop it into Matillion and it should run with little to no adjustments to the code. Here is a decent doc of how you access Blob storage with Python: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python
With that said, depending on what you are trying to do, you may want to pass variables into your Python script. Using variables in Python should be explain in the Matillion documentation. Unfortunately, I am not able to get to the Matillion docs right now or I would send you a link.
If you have a specific example of what you are trying to do, I would post it as a new thread. It's possible that there are multiple ways to tackle your scenario. Hopefully this helps in some way.