Thanks for sharing @MichaelBoucht . Especially regarding reset of the ids in the runhistory this leads us to extend the key, too!
@MichaelBoucht, I agree with @Bodo, the tip on id resets for job history is great to know. We offload the job history data nightly and have some dashboarding around that data to help us visually see things like failures. We didn't catch that the id's were resetting after a migration.
If you are on AWS I would highly recommend using the CloudFormation templates. You can do a ton in the way of the migration process within a CloudFormation template. It's really unfortunate but this is an area with the Matillion product that you get out what you put in since the process is so manual. Automation goes a long way in this regard. As an example of how powerful this can be, with a CloudFormation template and a couple Python scripts, we can deploy a new instance migrate to the new instance, shut the old instance down, and switch the subdomain name in Route53 to point to the new instance. The migration process literally takes 15 mins and only requires us to kick off a deployment from our Git repo. You don't have to but you can parameterize the heck out of the CF template as well which is what we did and this gives you the ability to handle things like deploying/migrating a single instance to a dev AWS account and a High Availability clustered instance to a production AWS account at the same time. Again, this is an area that you can put a ton of effort into speed things up or put a little effort and just make it better. Everyone's situation and need is different, which leaves a lot flexibility in the CF template approach.
With that said, I agree with everyone it really should be much much easier than it is without us having to do the heavy lifting. 😁
Thanks for contributing. It's much appreciated!
Hi @MatillionProductTeam ,
great job on the new document. I think this really helps us users to decide for an appropriate update method. The document is well structured and good to understand.
A minor remark for the "Cons" list of the migration feature: Task history is also not migrated and you will loose all previous job runs during the migration. Also - as Bodo Mentioned before - run ids are reset to start from 0 and thus it will even break your job history if you store that in an external place.
Thanks for working on this!
Michael
Hi @MatillionProductTeam ,
thanks for documenting best practices for updating Matillion. This is a good start for customers to make an educated decision on how to update and develop an own update strategy. Based on the current document I think it would be great to think in the direction of a more devops oriented approach regarding automation best practices (as @Bryan already suggested). Depending on the cloud infrastructure it would be great to have guidance on how to use tools like CloudFormation (AWS), DevOps Pipelines (Azure), Terraform etc. to drive the process. It might even be a good idea thinking about an open github repository that contains base templates to kickstart own developments.
Thanks @Bodo
This is brilliant feedback again, I will pass this onto the @MatillionProductTeam It is really valuable feedback for us, so again thank you for your support on this.
Kind Regards, Joe
Thank you so much @Bryan for this. We are on AWS and was thinking if CloudFormation would be the way. Still new on the whole AWS side so going to put some time on learning that whole area to grasp the architectual side a bit more better.
Your words give me confidence that that would be the right (one of the right) path(s) to solve this kind of issue.
Thanks again.
-Michael
Thank you @Michael
We really appreciate you reading over this and giving us further feedback, we will be sure include this in the document! its a great spot by yourself and @Bodo , We are really looking forward to hearing what other users think too, its been a great team effort, again thank you for helping us with it!
Kind Regards, Matillion Product Team
@shivani.bothra1620819956367 let me start with a caveat: my answer is based on experience from last year, there may be new capabilities in Matillion that may help. I can't comment on those, not having used them. Also, I'm assuming you're talking about doing a new instance upgrade since that was the context of my comment.
We didn't find a workaround to get the versions and Git information over 100% unchanged. You kind of have to decide if your version information or your Git information is primary and setup your new instance accordingly. This was a while ago now but I think you're basically looking at choosing between 1) migrating with the migration tools or importing jobs and then using that content to update Git; 2) cloning/checking out from Git to repopulate your versions.
Really the key is making sure that your Git and version contents are consistent with each other so that regardless of the method you choose, you're not losing any job content (just potentially losing some job metadata)
Hi @StanT (Customer) ,
Thank you for your response.
Did you setup versions manually and then populated from remote repo??
We migrated recently to the latest version of Matillion and approach we followed was just updated AMI id as we have external RDS instance already up and running.
We could see all the things in new version but this Git part is showing below error :
We are already checking with support in this regards.
Thanks,
Shivani