I am looking to integrate AZUREDEVOPS to Matillion...Any suggestions or steps on how to do?

Matillion ETL
1

I really appreciate any help or documentation of the steps for the integrating the matillion projects with GIT

 

What I am doing is export the project and checkin into GIT.

 

I really appreciate any help on integration of GIT with matillion to avoid manual steps.

2

Hi @DR838429​, great question. We don't use Azure but we do use a combination of Team Foundation Server as a build and release server against repos in GitHub and Git within TFS. Almost all of these uses are by App Dev teams and not by the data warehouse team. It sounds like you are trying to use your own method of Git implementation without going to the larger instance that includes the Git Integration. Is that correct? This isn't a bad thing by the way nor is it frowned upon in my opinion.

The problem with exporting a project in Matillion is that you only get the project. It doesn't sound like a problem until you want to include things like the Passwords in the Password Manager or the Shared Jobs which are dependencies in projects if you are using them. What I am getting at is that in most cases, just the project export is not enough. The other issue I have seen is that Matillion can add, remove, or change their JSON schema for exports depending on the version of Matillion you are using. Meaning, if you fast forward a year, you will likely be on a new version of Matillion and the old version of the project in Git may no longer work in the current version of Matillion. If they do work they likely will not work the same as they did when the project was pushed to the Git repo.

You may look at my post from yesterday about deployment from one instance to another (https://matillioncommunity.discourse.group/t/do-you-have-a-need-to-control-your-deployments-from-one-instance-to-another/1214). This still doesn't solve all the issues I posed above but it does get closer in that it would include all the dependencies. You could theoretically use Azure DevOps to execute a Python script that would do the export to files for the project group, project, shared jobs, API profiles, etc. then push those to the Git Repo. You could likely just take the export Python script and modify that one file and get everything you need.

I have had some conversations with Matillion about this topic and it's definitely something they are looking to improve upon. It's just something they don't have yet.

I hope this helps in some form or fashion. Let us know if you come up with an awesome way of doing this outside of the approach I threw out there. Thank you!

3

Hi @DR838429​ . We are trying to do the same thing. We had to connect directly to github in order to use it, as we received this error when trying to use Azure DevOps: Error while pushing

Caused by: [link] not authorized. We are currently using this approach: https://documentation.matillion.com/docs/2974180#

We will continue to investigate why we got a authorization problem and hopefully it is something we need to do on our azure devops side, but hope @MatillionProductTeam​ could give an official answer if this is supported or not : )

Ill update here if we find a solution : )

4

Thanks Bryan, Yes, thinking of any simple way of managing this with button click. Really appreciate your response. If I do anything different, definitely I will share here. 😀

5

Hi @DR838429​ ,

we also follow an own git based approach like @Bryan​ based on batch scripts, that do the import/export with all the point mentioned. One additional reason for us is, that we have an automated approach to version and migrate our database (DDL and data if necessary) and thus we have everything in one place and consistent version number for all different artifacts.

6

I am planning the same thing . use Azure devops and Azure pipelines to Do CD.

 

DEV branch-->pullrequest(approval)--QA branch --Approved then Automatically trigger pipeline to Fetch from QA Branch then same steps for Prod .

 

 

something like this ....

0694G00000M0hrSQAR_0D74G000008QbXBSA0
0694G00000M0hrSQAR_0D74G000008QbXBSA0516×225 13.8 KB

 

7

0694G00000LUsWwQAL_0D74G000008NGdCSAW
0694G00000LUsWwQAL_0D74G000008NGdCSAW1187×591 35.3 KB

Here is our Idea with Azure Devops . We are yet to implement it.

 

For Every Project, we create an Azure Devops Git Repo and 3 or More branches . and an 2 Azure Pipeline s (one for QA and one for Prod ).

 

  1. Developers Commit Code to Dev Repo;s
  2. Create pull Request (PR) to Merge code to QA (Silver Branch)
  3. PR gets approved by approver and code mergers to QA (Silver Branch)
  4. Create an Azure pipeline that Trigger automatically based on QA (Silver Branch) and run a matillion GIT /SCM Api’s a Fetch and commit it to matillion (QA)
  5. Same Process continues for Prod .

 

8

Hello @MN098168​

first of all thank you for a really good thread to read through, we took this away and looked into it for you.

Azure Devops has a remote git repository and, as such, we can push changes from Matillion as is the case with bitbucket.

How does Matillion integrate in a CI/CD Pipeline such as Azure DevOps?

Azure DevOps does have the ability to manage Git Repositories and Matillion’s Git Integration will work with it as a Remote Git Repository. As such the basic functionality should work as documented.

From an Azure DevOps perspective if the remote git functionality offered by the Azure service is simply a remote git with some wrappers, then the majority of our documented git integration should work as expected. This is in a similar thread to using either bitbucket or github, it shouldn’t really cause any conflicts.

Please find the aforementioned documentation also Git Integration & Getting started with Git and Azure DevOps: The Ultimate Guide

We hope that helps and please do keep us posted how your work goes!

Kind regards, Matillion Product Team