I have a project in which I need to complete the following:
Generate a query to get a data set.
Create a CSV file containing the data set mentioned above
Upload this CSV file daily to an SFTP
We are currently on Version 1.51.8 so we don't have new components to make this easy.
the query is pulled from Snowflake for the record
I'd try the following approach:
1) Create a table or view with the query result in your Snowflake DB
2) Use S3 unload component to create a CSV file in your S3 bucket
3) Use the Data Transfer Component to upload the file from S3 to your target SFTP server
how do you handle the multi part load of the file into the S3 bucket? I'd rather there just be one file. My dataset is fairly small with around 3000 records.
HI @kerri.gathright ,
To ensure it outputs a single file you would need to set the "Single" property to True. I can't remember if Snowflake requires you to set the Max File Size when you use Single=True or not but if it errors, you may want to set that property as well. Hopefully this helps!
this is exactly what I was looking for, thank you so much!