I want to hide a password in the python component without using the environment variable. I stored the password in Password Manager but couldn't pass it to Python like the normal variable '${var}'.How can I achieve this?
Hi @ganesh_pitchai ,
Unfortunately, retrieving passwords and making them available in Python is a bit of a conundrum with Matillion in it's current state.
You could leverage the Matillion API to get the password from the Password manager but that comes with some caveats. If you wanted to do that via Python you would have to supply the username and password for the account that has access to the Matillion API. So, you are back to dealing with clear passwords in Python. Another approach is to use an API Query Profile/Query Extract Profile with the API Query or API Extract component which would retrieve the password using the Matillion API. The problem with the API Query and/or API Extract components is that it only offers the ability to take the data and load it into a table. Theoretically it would work because you could then go query the table that the password landed in, put into a job variable, then drop the table. This is a lot of work to get a password!
I know Matillion has some things in the works around supporting password managers besides their own but I am not sure as to when that will be become part of the product. Since it sounds like you are on AWS I would look to leverage the AWS Secrets Manager. You can easily access the Secrets manager using Python to retrieve passwords and it's very straightforward. This is something we have pondered for a while since our App/Dev teams use the Secrets Manager for storing usernames and passwords. If you need a Python example of how to pull information like passwords from the Secrets manager let me know. I have a snippet of code floating around that I can go find. I hope this helps. Thanks for posting and have a great weekend!
The actual problem is we are migrating Jython to python3 When we trying to connect redshift we need to create the redshift connection here to perform the cursor=con.cursor(). We are feeling bad to store all credentials in an environment variable.
Can we achieve any way?. Thanks.
Hi @ganesh_pitchai,
Yeah, I would feel the same way. I don't like credential information bring stored within variables either. This is an example of getting credential information from the Secrets Manager in AWS:
import logging, boto3
from botocore.exceptions import ClientError
def getAPICredsFromAWSSecrets (region, secretsNamespace):
# Get the secret for the the Matillion API service account for use later
session = boto3.session.Session()
client = session.client(
service_name='secretsmanager',
region_name=region
)
try:
response = client.get_secret_value(SecretId=secretsNamespace)
except ClientError as e:
logger.exception('\t' + str(e))
return()
else:
return response['SecretString']
#Call the function and supply the region and Secret Name that you want to pull the value for
creds = getAPICredsFromAWSSecrets('us-west-2', 'YourSecretNameGoesHere')
#Pring the Secret value
print(creds)
Another thing to note is that the value for the Secret doesn't need to be just a password. We actually store a JSON object that has the username and password information in the Secret Value. That way the no username or password is ever exposed via variables. Here is an example of what the Secret value could be:
{
"username": "MyUserName",
"password": "MyPassword123"
}
When you retrieve the value from the Secret using the function above you would then be able use them in a format like creds['username'] and/or creds['password']. It works quite slick.
Keep in mind the Matillion Role will need access to the Secrets Manager or you will get a message indicating that it doesn't have permissions. I hope this helps explain how this can be used.