I would like to load 4 CSV files from S3 bucket to 4 different Snowflake tables. All loading tasks are independent and csv files are available in S3 Bucket like /basefolder/subfolder/<4csvfiles>

Matillion ETL
1

I have used Fixed Iterator, Variable='v_tablename' and Iterator values are TAB1,TAB2. Fixed Iterator component got validated and is fine. Now I am using S3 Load component. My question here is can I use Target Table as ${v_tablename}? S3 Load component is not getting validated. I would want to use only single S3 load and an iterator. Could you please suggest if the above steps are right way to do or any references to achieve this.

 

Thank you

2

Hi @KalyanB​,

Yes, that should certainly be possible. In your Target Table component tick "Use Variable" and then choose v_tablename from the dropdown.

0694G00000GGg8oQAD_0D74G000007jgopSAA
0694G00000GGg8oQAD_0D74G000007jgopSAA858×355 22.8 KB

A couple of notes:

  • You'll presumably need the Fixed Iterator to loop through two variables: one to hold the CSV name, and one to hold the table name
  • Matillion uses the default value of variables to validate. So if you have an empty default it won't validate (although it may still run perfectly fine). Please see this document for more information.

Best regards,

Ian

3

I even read the below article. Hopefully this should work in the current version of Matillion I am using by installing Python packages as and when required through Pip.

 

https://exchange.matillion.com/s/exchange-job/a074G00001Ek73jQAB/pip3-install

4

Any other alternate ways of approach are welcome.