How do i preserve the newlines in the job variable value?

Matillion ETL
1

i have text data with newline and i want to preserve it when i pass the variables down the jobs.

2

Hi @Joel​

This has raised some really interesting internal discussions, I just wanted to get a bit more information on your process so far.

How are you setting the variable is this being done with python or have you tried to input this into the UI, if so have you had any problems / issues?

I appreciate that is a few questions in one there, it would be really interesting to understand our current progress and setup.

Kind regards, Joe

3

Hi Joe,

 

Basically i have a job which is fully parameterized. All the metadata is stored in a table on Snowflake.

 

I am having a environment variable which holds a base64 encoded RSA Key which i need to pass to my parametrised SFTP Data Transfer Component.

 

I am using Python to decode the value and get into a proper format (new lines / LF) but when i assign it to the job variable it looses the newlines (LF) and after adding value in the properties it gives connection error.

 

i tried to search some javascript code (atob function to decode the base64 ) but it doesnt work at that level.

 

My SFTP job is like plug & play, any new connections with username password stored in matillion env or password manager will be picked up from metadata and file transfer will happen to desired location in the Buckets containers.

 

But right now i have to duplicate the job just because the RSA key has to be hard-coded in the component

 

Solution i thought was storing the RSA key in Instance.

4

Hey @Joel​

Thanks for sharing that information with me, we have seen users in the past create Python 3 script task's which assign the RSA key to a variable called "Key" it prints it to the console and assigns it to the job variable.If you are still having some problems achieving this, I would recommend raising a support ticket, our dedicated solution architects will be able to support you with this.

Kind regards, Joe

5

This works for me.

 

Thanks

6

Brilliant @Joel​

Our users are often sharing their user stories on the community, I am glad this one helped you achieve what you wanted 😊

Kind regards, Joe