I'm trying to use a File iterator to accomplish this but having trouble connecting to the container. When I put in the path I keep getting this message 'Parameter this.client.subscriptionId() is required and cannot be null.' What am I missing in the path? Is there a better way to accomplish this? Any help will be appreciated.
Have you granted your Matillion VM the privilege to access any of your Storage Accounts? Matillion don't seem to have much official documentation on it, but I think you need to assign a role on the Storage Account to the VM. You have to log into the Azure portal and follow this Azure guide. I normally use the Owner role and that works OK.
HI @DF214566,
What Cloud Database system are you trying to create a table per file in? Snowflake? There are several ways of accomplishing this.
If you are using Snowflake, the best and most universal way to accomplish this by using native Snowflake functionality. The idea would be to ingest each file into a raw table via a COPY INTO statement. With that you can automatically get extra information about each row. The columns would look something like this: file_content as variant, file_path as string, load_datetime as timestamp. With that you can just write some SQL that does a "select into table_name" where the table_name is the name of file. You can facilitate most if not all of this via Matillion using the SQL Script component. I hope this helps in some way. Have a great weekend!
Thanks @Schepo. Do you need to be an Azure admin to assign a role on the storage account?