Hi all,
Scenario is: I have the sorted data in a bigquery table but the matillion "Cloud Storage Unload" component is randomly unloading the data in a file. We don't have any sorting functionality in matillion "Cloud Storage Unload" component.
I cannot use python script as required packages are not present and i am not allowed to install one. Avoiding Bash script as it has its own default service account related issues.
Does anyone know how can i do that?
Hi,
I'm encountering the same issue with S3 Unload. I'm currently using a Python Script component to address this, but it's taking a considerable amount of time to complete the task.
Does anyone know if it's possible to obtain a sorted file using the S3 Unload component?
Thanks!
Hi Claire, I was working with Amazon RedShift. I've found a better solution to the problem. I created a grid variable and then used a table iterator with sorting in an append to grid component. Afterward, I used a Python script to take this grid variable and write the file.
It works better than iterating directly over a Python script. When I tried to iterate directly over a Python script, it was too slow, as I mentioned before. So, if the table had more than 1000 lines, it didn't work well.
How I solved it, i stored the data in a view in sorted way and then unloaded the data using "Cloud Storage Unload". It actually stored the data in sorted way.