What are various CI/CD tools that can deploy matillion jobs?
Here is our Idea with Azure Devops . We are yet to implement it.
For Every Project, we create an Azure Devops Git Repo and 3 or More branches . and an 2 Azure Pipeline s (one for QA and one for Prod ).
- Developers Commit Code to Dev Repo;s
- Create pull Request (PR) to Merge code to QA (Silver Branch)
- PR gets approved by approver and code mergers to QA (Silver Branch)
- Create an Azure pipeline that Trigger automatically based on QA (Silver Branch) and run a matillion GIT /SCM Api's a Fetch and commit it to matillion (QA)
- Same Process continues for Prod .
High-Level Flow.
Here is the code:
save as matillion_cicd.py
# About : matillion API to fetch code from Selective branch and commit to matillion instance
# Author: Ranjith
# Date : July 2022
# Documentation : https://documentation.matillion.com/docs/5971755
import sys
import json
import requests
from base64 import b64encode
import warnings
import os
warnings.filterwarnings('ignore')
server_name = sys.argv[1]
api_version = sys.argv[2]
proj_group_name = sys.argv[3]
proj_name = sys.argv[4]
mat_username = sys.argv[5]
mat_password = sys.argv[6]
git_username = sys.argv[7]
git_secret = sys.argv[8]
git_branch = sys.argv[9]
try:
if all(v is not None for v in [server_name, api_version, proj_group_name, proj_name, mat_username, mat_password,
git_username, git_secret, git_branch]):
print('All parameters are passed to the script! \n')
else:
raise Exception
except Exception as e:
print('Not all parameters are passed, pls check!')
sys.exit()
scmfetch_url = f"https://{server_name}/rest/{api_version}/group/name/{proj_group_name}/project/name/{proj_name}/scm/fetch"
scmGetBranchId_url = f"https://{server_name}/rest/{api_version}/group/name/{proj_group_name}/project/name/{proj_name}/scm/getState"
scmCommitCode_url = f'https://{server_name}/rest/{api_version}/group/name/{proj_group_name}/project/name/{proj_name}/version/name/default/scm/switchCommit'
matillion_headers = { "Authorization": "Basic {}".format(b64encode(bytes(f"{mat_username}:{mat_password}", "utf-8")).decode("ascii")),'Content-Type': 'application/json'}
def Matillion_migration(scmfetch_url,scmGetBranchId_url, scmCommitCode_url,server_name, api_version, proj_group_name, proj_name, mat_username, mat_password,
git_username, git_secret, git_branch):
fetch_payload = json.dumps({
"auth": {
"authType": "HTTPS",
"username": git_username,
"password": git_secret
},
"fetchOptions": {
"thinFetch": "true"
}
})
# To perform a fetch from a remote repository, make the following POST call:
scmfetch = requests.request("POST", scmfetch_url, headers=matillion_headers, data=fetch_payload, verify=False)
if scmfetch.status_code==200:
print(f'Response from fetch {scmfetch.json()}')
print('\n')
print('----------fetch is Completed!----------')
print('\n')
else:
sys.exit('Error at scmfetch, Status code is not 200')
scmBranch_payload = {}
# To retrieve a list of each Git commit on the Remote repository, make the following GET call:
scmBranch_response = requests.request("GET", scmGetBranchId_url, headers=matillion_headers, data=scmBranch_payload, verify=False)
if scmBranch_response.status_code!=200:
sys.exit('Error at scmBranch_response, Status code is not 200')
for i in scmBranch_response.json()['result'].get('commits'):
if i.get('tags')[0].get('text') == git_branch:
ref_id = i.get('referenceID')
print('Git_Branch passed is: {}'.format(git_branch))
print('Corresponding Git_referenceID is: {}'.format(ref_id))
print('\n')
break
print('----------scmBranch is Completed!--------- \n')
scmcommit_payload = json.dumps({
"commitID": ref_id })
print('scmcommit_payload: {}'.format(scmcommit_payload))
# To switch to Git commit using the API from the selected branch, make the following POST call:
scmcommit = requests.request("POST", scmCommitCode_url, headers=matillion_headers, data=scmcommit_payload, verify=False)
if scmcommit.status_code!=200:
sys.exit('Error at scmcommit, Status code is not 200')
scmCommitCode_json_response = scmcommit.json()
return scmCommitCode_json_response
if __name__=='__main__':
json_response = Matillion_migration(scmfetch_url,scmGetBranchId_url, scmCommitCode_url,server_name, api_version, proj_group_name, proj_name, mat_username, mat_password,
git_username, git_secret, git_branch)
print('Final response: {}'.format(json_response))
------------------------------------------------
run with arguments
matillion_cicd.py $(Mat_Server) $(Mat_API_Version) $(Mat_ProjectGroupName) $(Mat_ProjectName) $(Mat_User) $(Mat_Secret) $(Git_User_Name) $(Git_Secret) $(Git_Branch)
Updated CICD Script
run with arguments
matillion_cicd.py $(Mat_Server) $(Mat_API_Version) $(Mat_ProjectGroupName) $(Mat_ProjectName) $(Mat_User) $(Mat_Secret) $(Git_User_Name) $(Git_Secret) $(Git_Branch)
can you provide the steps how to implement cicd on matillion project
Hi @Ranzith , We are thinking of a similar approach but using Github Actions.
Just wondering , is creating pull requests at Step 2 and Step 6 , are manual processes ? or Automated processes in Azure Devops?
Cheers,
Anu
Hi @Ranzith were you able to implement CI/CD using Azure DevOps? Also, could you connect to Azure Repos directly using Matillion Clone Remote Repository option?
Hi Ranzith,
I had few queries
1) Is there no true CI or build process that pushes the code built code to Artifacts?
2) Does the branching strategy always needs to be aligned to the environments (in other technologies we have one release branch code which is deployed to Test1, Test2 and Prod sequentially as per deployment pipeline setup)?
3) Have you implemented rollback strategy?
Regards,
Vaishali
Thank you
Thank you @Ranzith
Hello @apalla1.miraclesoft
Thanks for your post, I have some documentation here that should be able to help you with that.
Please keep me posted on how you get on with that.
Kind regards, Joe
Yes it’s a Manuel process or you can automate as well . In our case we want to review code by lead or architect before moving to next environment.
you can automatically trigger for non prod env’s but I Recommand approvals for prod .
Yes we are able to Implement Azdo from end-end .