Thank you for raising these ideas and again thank you to all the users who up-voted and ensured they were added to the product roadmap and will be implemented.
You will all receive a community badge to reflect this 👏
And please do help us shape the Ideas Portal, we have a feedback survey ending soon here
large Enterprises don't want to go thru the process of getting new servers for every upgrade. the recommended process is what I won't agree with as well as the migration is not easy, users have to be migrated from one system to another, there are issues with firewalls rules, etc ...its not easy to work with enterprise Cloud teams explaining this scenario ... getting new hots every time when there is a new upgrade. We think the cloud is easy just a new AMI with a new version and Ec2 but in large enterprises, it's not the same. We have to go thru a lot of processes ...
The main reason to avoid the "freh instance" approach is the included migration tool which lacks some features. For example, user accounts and project permissions are not migrated. Also, the GIT configuration is not transferred to the new instance and GIT has to be set up from scratch on the new instance.
I think what millions did with yum updates is good you have to formalize it and make it work in such a way we don't need new servers each time we upgrade.
Finally, I wish there is a SaaS UI for the front end.
I can't find any of the four implemented ideas in the release notes. I always though that the release notes should cover all changes in the product. But if those four things are already missing, what else has changed in 1.54.7 that is not mentioned in the notes?
Over the last 1.5 years I have done several Matillion upgrades with various setups and configurations. I have learned a ton about what works and what doesn't. Although I agree with the pain of upgrading EC2 instances via an AMI and the handful of challenges it brings, I think the approach makes all the difference in the world. We have some AWS experts with 5 or more certifications under their belt. Even though they understand AWS extremely well, I found that unless they understand how Matillion is assembled from a database and Ec2 perspective, their solutions to upgrades resembles something of a brute force approach. There are some best practices and essentials that every Matillion Admin should know about upgrades which will make upgrades about 10x smoother with less heavy lifting.
Items that have helped us:
Separate the Matillion database from the instance (THIS IS CRITICAL TO MAKING THINGS SIMPLE)
Postgres RDS is your friend in this scenario
If you do this, you no longer have to worry about migrating anything or losing Git configurations, Passwords, etc.
Use CloudFormation Templates even if you are using non-clustered Ec2 instances
Allows for a repeatable deployment with the same outcome every time
All your Matillion IAM Roles, Firewall rules, Route53 configurations, subnets, etc. can be done in a matter of a couple minutes which ensures Matillion has everything that is needed when it comes up
Create a subdomain for your Matillion instance in Route53
Doing this allows you to automate the process of pointing the subdomain to the new instance as it comes up
Use Auto Scaling Groups to help control instances during upgrades
Auto Scaling Groups will allow you to cycle in a new upgraded instance when you are ready rather then deploying everything only to find out something went wrong and now your Matillion instance doesn't work at all
By employing the above tactics, I am able to deploy an upgraded Ec2 instance with the latest updates in about 15 mins with a total of 5 mins of Matillion down time.
The last thing I would say is that I would be very careful of using YUM for updates across major versions. You will never know if Matillion has changed the product to use a specific version of Tomcat or to look for a specific configuration in Linux until your instance stops working after doing an update. To put it simply, only Matillion knows what Matillion requires for each of it's versions. If you are deploying a new AMI as a new instance then you know all dependencies are accounted for and tested by Matillion to be working. A yum update of Matillion doesn't necessarily upgrade Tomcat when there is a potential security vulnerability in Tomcat. You don't necessarily want to arbitrarily upgrade Tomcat either as this could break Matillion due to some dependency configuration or functionality required. Upgrading Tomcat is unlikely to cause an issue for Matillion but there is a chance and should be considered before upgrading Tomcat if you choose to do so.
I hope this info helps others in their Matillion upgrade journey.
Thanks so much for your feedback on the migration tool. It is certainly an area we are looking to improve as we know it is currently difficult to migrate. In the short term, we plan to plug some of the gaps we know are there in the existing tool such as user migration and git, however, we know there is still a long way to go to make this simpler, more seamless experience. In the longer term, a SaaS tool to manage migration etc is definitely a direction we are looking at so watch this space.
Hi @Ranzith, if you already have your Postgres DB separate from your Ec2 then you are already in a pretty decent place. I will say that 1 thing made a huge difference on the ease of upgradability. To be honest, it took us several iterations to get Route53, Autoscaling, etc. setup. It's interesting that your company doesn't allow CloudFormation templates to be executed under a service account which only has permissions to specific actions but would be ok with a SaaS solution where they have no control over what the SaaS solution does on their backend. To each their own though...
You might gain some traction if you let them own the Cloudformation Template and the deployment process. Meaning, they would have full control over how the Autoscaling, Route53, Ec2, etc, is setup. This is sort of what we do although I have enough knowledge now and a close working relationship with the infrastructure people that they allow me to do most of the change on my own as long as they know what changes were added to the the CF template. I usually communicate the changes and get their sign-off before I deploy the CF Template.
To that point, I would recommend dropping your CF Template in to source control if you can. It allows you and anyone else with access to track every single change made to the template. It also gives you a way to roll back changes.