I have a GCP bucket with hundreds of csv files in the format [filename]-YYYYMMDD-[suffix].csv I want to fetch just the last 3 days' files each day to merge into BigQuery

Matillion ETL
1

Wildcards and variables in the URI for the Google Cloud Loader don't seem to appear valid. How would you approach this?

 

We are running Matillion on GCP, target is BigQuery.

2

Not sure about GCP, but we tried a similar thing on AWS. Unfortunately, File Iterator doesnt have enough filter options to limit the list of files from a directory other than pattern. You can try using a python script and filter the list of files you require (based on your Last Modified date) and then submit to your File Iterator to fetch them.

3

That's disappointing. The pattern doesn't seem to be recognized anyway, and I don't have the python knowledge to write such a script.