The S3 load component seems to do a recursive scan of the whole bucket instead of just returning the top folders. This leads to a very long validation time - and it doesnt list the folders anymore. It does this with each validation, ending up in delayed schedules etc.
We're able to reproduce the issue on a fresh new EC2 & redshift cluster with all rights granted. Support wasnt helpful until now thats why this warning comes up here...probably someone else faced the same issue and had luck in resolving it. We tried nearly everything on AWS side as well as in the matillion config.
We are also seeing issues with 1.74 and S3 file handling and I can confirm the long validation times and run times.
I have to add another warning: In our case, the Excel Component was no longer able to handle spaces in file names. After the update, our team had to spend some hours to rename all the Excel source files and get rid of all spaces in the file names.
I'd recommend to test this update very carefully in a testing environment before updating any Production instances.
We are definitely experiencing this same issues since we upgraded to 1.74 on 1/24/24. When I use the data transfer from ftp to S3 OR network to S3, the 300 files that we move took 24 mins before the upgrade, now it takes 4 hours. Also, I get this error when using data transfer using S3, they removed the "log delivery write" option and I do not see that in the upgrade documentation. Also validation when you make a change is super slow AND my S3 folder are no longer listed.
Parameter Validation Failure:
Access Control List Options - Unrecognised options: Log Delivery Write
We're experiencing the same issues (not being able to browse the S3 bucket, the validation of the S3 Load component hanging and never finishing) as well. We've tried 1.74.0 and 1.74.3, both versions were affected. We've reported this to Matillion Support and we're hoping for a quick resolution as it completely blocks us from updating our instance.
1.74.4 fixed the S3 browsing load but did not resolve the Data Transfer failure (The specified key does not exist. (Service: S3, Status Code: 404) during the orchestration component task when loading/unloading to S3 despite no difference in IAM permission or Matillion configuration.
It's really positive to see your input on this thread -- it feels to me it would still be worth you looking into getting some answers from your engineering team and sharing them with us, rather than us trying to figure this out based on one another's experiences after patching our instances.
The Release Notes for the latest patches include the following fixes which sound relevant:
Fixed an issue where processing AWS S3 object lists was not working as expected. (1.74.4)
Removed an AWS call to request the region for S3 buckets. (1.74.2)
However, from this it doesn't sound very clear to me exactly which issues with Matillion components that we saw after the upgrade have been resolved with those fixes.
Based on other people's responses, browsing the S3 bucket is fixed -- is the long validation of the S3 Load component fixed as well (sounds like it might, but it's a guess)?
Are there any outstanding S3 issues that your team is aware of but for which fixes haven't been rolled out?
Updating our instance requires quite a bit of co-ordination between our data teams, so we're unwilling to try another update until we're confident we're not going to run into problems again.
I just wanted to get a full update for you all with regards to the S3 issues you have been experiencing, I can confirm this is a priority for our team and they are working hard on it as we speak.
A number of fixes have gone in to remedy this and are currently being tested, the process for this is quite stringent as there are many permutations of bucket setups to test against.
If this testing is successful we will be pushing a quick release of 1.74 to include them.
We apologies for the delay and blockers this has caused, but rest assured the team are working hard to get this fixed.
I hope you had a nice weekend, the team are close to finishing testing on the fix for these issues and a release should be very soon, as soon as this released is fixed, I will let you know here.
I appreciate it's taken a little longer than we had hoped, but we are very close to a resolution.