MATCH_BY_COLUMN_NAME does not work

Data Productivity Cloud
,
1

We are in the middle of migrating from Matillion ETL to DPC on a Snowflake platform. One of the (few) new changes that we were excited about was the introduction of the “Match By Column Name” option that is now included in the S3 Load component.
However, this option does nothing. The parameter is not pushed down to Snowflake, even though running multiple times and changing this setting.
Could you please give insight how this can happen?

2

Hi @scwin813

Thank you for your post and for sharing this issue you have encountered. I shared the information with the team, and it looks like this is an issue that we have now raised, and the team are investigating. As soon as I hear more, I will give you an update.

Thank you for bringing it to our attention.

Kind regards, Joe

4

Hi @scwin813

Thank you for your patience on this, I wanted to check that your set up followed the following guidlines set by Snowflake:

For a column to match, the following criteria must be true:

  • The column represented in the data must have the exact same name as the column in the table. The copy option supports case sensitivity for column names. Column order does not matter.
  • The column in the table must have a data type that is compatible with the values in the column represented in the data. For example, string, number, and Boolean values can all be loaded into a variant column.

This is taken from the documentation here

I have also been informed this will not work if it is a CSV or XML.

Let me know if this information changes anything and we can continue to look into it.

Kind regards, Joe

5

Hi Joe,

Ah, well that does explain it since we are loading from CSV files. It is supported in Snowflake, and the property is mentioned in the documentation so I assumed it would work. It might be a good idea to add a disclaimer to the documentation for the S3 Load component, to avoid futher confusion for others.

Is this functionality on the roadmap for CSV files?

Thanks for your reply!

2 Likes
6

I agree. I should have mentioned that I have sent the feedback over to our documentation team to reflect this.

I don’t see that on the roadmap or the ideas, Please feel free to raise it here.

Let me know if I can help with anything else.

Kind regards, Joe