Best Practice for Migrating Jobs to Project Versions

Matillion ETL
1

I am trying to leverage Matillion's built in versioning to handle source control for jobs. I've done the walkthrough materials of creating a Production version and seeing that I can make a change to a job in Default and it not reflect in Production. My question is - how do I push a new change from Default to Production?

 

For example I have job A and job B both in Default and Production. I make changes to A and B but only the change to A is ready for Production. Do I need to rename Production to something Old Prod, then create a new version called Production New based on Default? If so how I can make it so only job A moves over to Production New and not the in dev job B?

 

I was hoping to use the internal versioning because git isn't required here and seems overly complicated for what we need to accomplish. Should I bite the bullet and focus on integrating git successfully?

2

Hey @CFitz​

Looks like the community have been unable to answer this one, its a great question and one our solution architects will be more than happy to answer for you, if you send over a support email they will be able to advise you on this. Please do share your findings here though, myself and the community would love to hear how you managed this.

Kind regards, Joe

3

I have the same question and am curious to hear what advice was shared from support to @CFitz​ ?

We tried using the Matillion GIT integration for deploying from Dev to Test and then to Production as outlined in the help article 'SCM Integration' at https://documentation.matillion.com/docs/2871146

Similar to what @CFitz​ outlined above, only some of our changes were ready to deploy (typical scenario for us when deploying) but we found that ALL changes were being deployed. We had to abandon that method of deployment and instead have continued with our more manual method of exporting jobs from dev/test and then importing them to test/production.

4

Hello, I have no experience using the Matillion in-built versioning, however, we have our process as the following:

Two instances of Matillion:

1 Production

1 Development / Test

 

Production instance has only one project - Production.

 

Dev instance has 2 projects, Development and Test.

 

We use AWS CodeCommit (GIT) to deploy changes into higher levels.

 

In Development, we work our magic, and commit our changes into a branch in git, called 'Development'.

We push this change into CodeCommit.

 

Swap project to Test, pull changes, let the testing begin.

If all is fine, we can merge the changes into master-branch. (or another branch that is used by Production, in our case master).

 

Push the merge into CodeCommit.

 

We go to Production instance, we pull, set master branch latest or most appropriate as active, and deployment is done.

 

There's a bit of nuisance when something gets deployed all the way to prod and is required to be reverted, this usually isn't a problem if we would merge every commit one-by-one in the master-branch, then we would just 'redo' the merge without the problematic commit.

 

However, we usually fix it and do a new commit due it's quite fast do to with these fancy new tools anyways :)

 

I think that's it, feel free to ask anything related on this. Always looking for a second opinion and other/smarter way of doing things :)

 

-Michael

5

Hi @AKelly​ , thanks for your info, I do have in the same page, but I do have a quick clarification on manual import/export, when import job has the same name as an existing job, it will be given a numbered suffix when imported, how are you handling this issue?

Thanks in advance.

6

Hi Michael,

CodeCommit handles files, so I assume you export from Matillion to a respository of json files, and git-handle these files, and then import to the Test or Prod projects. If this is the case, do you have one file per job or per folder? When importing to Test/Prod, do you import the whole repository, or just what has changed?

Best regards,

Lars

7

Hi @srajagopal1583163496535​ we export from DEV only the jobs that we want to deploy to TEST. Then, in TEST, before importing those jobs we rename the jobs in TEST. E.G. if I changed job jobOne in DEV I would export it and then in TEST I would rename jobOne to something like jobOne_20210107_AK (i.e. adding today's date and my initials), and once I've renamed the job in TEST then I will import the jobOne. After we've tested that the changed job hasn't broken anything in TEST then we delete the old renamed job jobOne_20210107_AK. This is a very manual process which works for us, but we would like to move to a better process for this.

Note that before importing anything into INT we do also create a backup of INT by creating a version.

8

We follow a similar workflow to you, which is this:

In Dev the developer duplicates the finished object and names it <release_number> - <object> .e.g "R5 - Orch Extract"

To release, we search for objects prefixed with R5 and export them.

Import them into the Test Project.

So now in Test we have an "R5 - Orch Extract" and the current working "Orch Extract"

We add a suffix to the current working version, remove the prefix from the R5 version and we're done.

 

It's a very manual process which could EASILY be better if there was an option to replace objects during the import. I'm searching now through the forums to see if anyone has suggested it or for reasons why it can't be done.

 

9

Hello @lars.ritland​ ,

The process mentioned uses the matillion for push/fetch/merge.

So commit in Matillion, push to remote-git. (CodeCommit in this case).

Go to Test Project, fetch from remote-git. Merge the 'project local' 'master' git commit against the remote commit 'development'

Test it. if all good. Commit, Push 'master' into the remote git.

Fetch remote-git master and set as 'current' version. (no need to merge anything if production is ment to be using the merged version from test as master)

Hope this clarifies :)

I know several uses export as json using the API to git.

10

Thanks @MichaelBoucht​ ! I didn't realize you utilized the built-in git feature of matillion. Now it makes perfectly sense :)