S3 Load GPG Encrypted Pipe Delimited txt.gz files into the snowflake tables

Matillion ETL
1

Hey Everyone,

 

I am working on an ETL automation process that requires loading GPG encrypted files from s3 to snowflake.

 

What component I can use to decrypt the file on the fly and use the regular s3load component to load /append the data into the snowflake table?

​

Any help is appreciated.

 

 

Thanks,

Himanshu Mohapatra

2

We had a similar use case and we were able to use a Python component in Matillion, with the "python-gnupg" package to decrypt GPG files, and the "boto3" package to manipulate files in S3. Not easy if you aren't good with Python (I'm not). I don't know of an easier way, but I'll be following this thread in case someone else does!

3

Hey Kevin,

 

Thanks for your suggestion. Could you please explain a bit about manipulating files in S3? Does that mean, we need to decrypt the files and temporarily store them in the s3 bucket until the load is complete to snowflake and then drop those decrypted files from s3?

 

Thanks,

Himanshu