Receiving a Validation Failure. An error occurred during component validation error

Matillion ETL
1

Hi,

 

We are trying to upgrade from our existing Matillion Server hosted on AWS ( 1.37.5 ) to the latest version available on AWS marketplace for Matillion ETL – Snowflake ( 1.53.8 )

 

A separate Ec2- instance with the latest version is launched, keeping the existing Matiilion server running.

 

We performed the Migrate activity of the Project using the Migrate steps mentioned in the link below:

 

https://www.matillion.com/resources/blog/best-practices-update-matillion

 

On successful migration of the Project, we are now facing the issue to Run the jobs.

 

The AWS Credentials ( along with IAM policy is same as before and was working for earlier version )

 

Now we get the error as:

 

Validation Failure. An error occurred during component validation: A request was made to create an S3 client with missing credentials.

 

IAM Policy:

 

{

   "Version": "2012-10-17",

   "Statement": [

       {

           "Effect": "Allow",

           "Action": [

               "s3:ListBucket",

               "s3:ListAllMyBuckets"

           ],

           "Resource": [

               "*"

           ]

       },

       {

           "Effect": "Allow",

           "Action": [

               "s3:ListBucket",

               "s3:ListAllMyBuckets"

           ],

           "Resource": [

               "arn:aws:s3::: bucketname"

           ]

       },

       {

           "Effect": "Allow",

           "Action": [

               "*"

           ],

           "Resource": [

               "arn:aws:s3:::bucketname/*"

           ]

       }

   ]

}

 

Can you please provide any pointers on the same

2

Hi @persistentengteam​ ,

Is the IAM Policy in your post part of the error output or is this your actual IAM policy? One thing to check is that the new Ec2 with the new version of Matillion that you spun up has the same VPC, Subnet, IAM Role, and Security Group as your existing older Ec2. I have made this mistake in the past and forgot to set the IAM Role or Security Group and had various issues after that.

Since you are taking such a major jump from 1.37 to 1.5X, there definitely could be some different security requirements that would drive an update to the policy the IAM role being used by Matillion. The article you posted looks like it has good info and the method you are following is definitely the correct and safest approach. Since the article is a little older, I am wondering if other Matillion or AWS changes since the article came out have driven perhaps other required changes that are not reflected in the article.

If you are sure the Ec2 configurations are identical to the previous one and you are still getting the error, I would open a support case on this. Matillion is likely the only one that is going to know the little infrastructure details that have changed between those versions.

Post back and let us know what you end up finding. Thanks posting!

3

Hi Bryan,

 

Thanks a ton for your valuable inputs.

 

We tried with IAM Role and attached it to the Ec2 instance instead of IAM policy and that did the trick for us.

 

IAM Roles & Permissions (AWS) (matillion.com)

 

Once this was done, we had to change the credentials manager to point to Instance credentials in the Matillion environment.

 

Would like to post the subsequent errors and its resolution as well in this post, so it may come handy for others.

 

Error 1:

"Environments:

Communications link failure

 

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."

 

This error was observed due to the default TLS implementation used by the matilion JDBC drivers for mysql.

In our case we dont have the mysql over TLS and since its internal within the VPC, we disabled the same.

 

File Name: /usr/share/emerald/WEB-INF/classes/jdbc-providers.properties

 

{"name"   : "MySQL",

"driver"  : "com.mysql.jdbc.Driver",

"url"    : "jdbc:mysql://<host>/<database>",

"fetchSize" : "Integer.MIN_VALUE",

"limit"   : "limit-inline",

"prepmeta" : false,

"allowUpload" : true,

"ssl": "false", <<<<<<<<-------- this line needs to be added

"defaultProperties" : {"connectTimeout": "20000"}

},

 

 

Error 2:

Insufficient Permissions: Request had insufficient authentication scope

 

To resolve the above issue, we had to reauthorize the Google Drive API in matillion again and also verify the call back url.

 

 

Cheers !!!