Can Matillion ETL convert multiple CSV files to Parquet?

Matillion ETL
1

I have the following folder hierarchy in an AWS S3 bucket:

 

Folder: January/10 

 

Files: 16b516c0-8f2a-eabd-770a-b8bbc83c5859.csv, 16b516c0-8f2a-eabd-770a-b8bbc83c5859.csv, …

 

In other words, every folder represents a calendar day with several CSV files nside. 

I would like Matillion ETL to do the following transform: January/10 AsingleParquetFile.Parquet

How can I implements this in Matillion ETL?

 

2

Hello @vicsotnikov​

Thank you for your post, this is a great question. We have seen users achieve this in the past, it's dependent on your CDW on the process but we have seen users utilise an external table and an unload component.

I hope that helps, if you have found another method, we would love to hear it too!

Kind regards, Joe