I am trying to supply SFTP Key, username and password to a file iterator and data transfer component using job variables.
Username and password is getting resolved correctly but the SFTP Key is not getting resolved due to which we are getting "Exception: Error opening sftp connection"
How can we pass SFTP Key to the component using job variables.
Overall I am trying to get username, password and SFTP Key from AWS secrets using Pyhton2 and assigning to job variables.
If I print the job variable, the value is printed correctly and I am also able to use the printed value in job by hard coding it. So it seems that we are able to fetch the key properly but the variable is not getting resolved in SFTP Key field of the component.
Attaching screenshot of how I am trying to supply variable in SFTP Key field.
Your approach seems sound to me. Have you tried using a separate Python script to print the key after it has landed in the variable? I am wondering if there is perhaps extra whitespace, characters, or malformation going on as it's being pulled from the AWS Secrets and stored into the variable. That's just the first thought that came to mind.
As per my opinion private key is a secret and hence it shall be treated in same way as passwords. Putting private key as a plaintext which than can be viewed/retrieved is a security risk. I consider this as a design mistake on Matillion side.
Private key shall be taken either form Password Manager (same as password) or "SFTP Key" field shall accept ENCRYPTED key + key password (also taken from Password Manager)
I am having the same issue. When I paste the plain-text private sftp key then the Data Transfer component works.
However when I use a variable ${sftp_key} then the Data Transfer component doesn't work. I noticed there is no option to use a variable either. It's plain-text or nothing apparently.
Hey there, yes I will attach the script here. I wanted to post it to the Matillion Exchange but the exchange is looking a bit strange right now and I can't see the buttons to upload new Shared Jobs.
Anywho, check over the python script that's in that job and you can modify it to suit your needs. It has a few python modules referenced so you may need to install those on your server if you don't have them already.
