Can u u suggest
Hi @DJ789446,
Are you new to ETL/ELT work or just Matillion? Also, which Matillion product are you using (i.e. Matillion for Snowflake, Redshift BigQuery, Databricks, Synapse, etc.)? Also what cloud platform are you on (i.e. AWS, GCP, Azure, etc.)?
Check out Matillion Academy for guides and trainings. They also offer guides on the beginner level.
Matillion snowflake and was cloud
Aws cloud
Are you new to ETL/ELT type work or just new to the product? This will help in being able to give good places to start.
Even if you are not new to ELT type work you will still need to understand the Snowflake fundamentals. So, let's ignore Matillion for a moment since Matillion is just submitting SQL statements to Snowflake to execute based on your configuration of a component that touches Snowflake.
Starting from the Extract and Load side of things, it would be beneficial for you to understand what Stages are, and how to create and configure them. From there I would start getting familiar with the Snowflake loading patterns. The tried and true and most common method is to ingest data from an S3 bucket where the data is in JSON, CSV, Avro, or Parquet file format. In most cases you are going to want to land that data in a table with the content of each file being inserted into variant column where 100% of the raw input will be. From there I would start looking at File Formats and the COPY INTO statement as these are required to ingest data from S3 to a Snowflake table. You don't need to know every option in the COPY INTO statement but you should know what it's for and how it can be leveraged to load or unload data. Knowing this will allow you to understand what Matillion is doing, why it needs specific propertied for components, and what the format of those properties should look like. Simply put it just take a lot of the guess work out.
Long story short when you configure a component in Matillion like the "S3 Load" to load data from files in S3 into a Snowflake table, Matillion is building the "COPY INTO" statement to send to Snowflake which will load the data.
This is the basics of loading data from files in S3. From there you can get into loading data from API's or other cloud systems which is where a lot of the power of Matillion comes in. Once the data is loaded you will need to start thinking about transforming data. Meaning you just landed data into a Snowflake table in a very raw JSON, CSV, etc. format, now what do you want that data to look like when you expose it to applications, reports, other systems, etc. The answer to that question will determine what you need in the way of transformations.
Take a look at Matillion's Youtube channel. Even though a lot of the videos are fairly old they are still very relevant. Either way, the Snowflake Documentation, Matillion YouTube videos, and Matillion documentation are good places to start. I hope this helps!