Hello,
Has anyone encountered Matillion GIT Integration "Walk Failure"?
Please see attached screenshot
Hi @DN536026
Sorry to hear you have encountered this, if you can try migrating the project excluding Git followed by cloning the repo into the migrated project, that should resolve this issue.
Please do let me know how you get on.
Kind regards, Joe
Per Matillion support team, "This may be related to a known bug. This could occur if a repository is corrupted. To resolve the issue, we deleted and re-create a new project, and clone the repository. The drawback of this approach would be that any changes that have not been pushed to the remote repository will be lost, but it should fix the issue."
Few things to remember:
1) Backup your environment variables
2) Backup your Environment (I.e. Snowflake Config)
3) Backup all your versions
Thanks for sharing that with the wider community.
Please let me know if that resolves this for you.
Kind regards, Joe
Oh yes, the infamous "Walk failure" message... We have ran into this many times before.
Where we see this message is right after we have migrated to a new instance of Matillion. What I believe happens is that once a project is migrated from an old instance to a new instance there is no local repo for the project on the new instance and it gets confused as to what commit it should be on. This only happens on projects where there were changes in projects that were not committed before the migration. The way we have fixed this in the past is to export the whole project to a json file. Create a new project and import the json back in. Then hook up the Git repo again and delete the old project. Keep in mind the import and export method doesn't get objects like the password manager content. You will need to add those manually to the new project. If you open a support case on this and they give you an answer other than mine, please do share. I would love a better solution!
Thanks @Philipp,
I wanted to try this solution but I am unable to since I am just an admin, not an actual users with credentials in those repos.
What do you think is causing this? Is there a size limit that the migration tool is hitting?
Can you see any other way of migrating the git files in bulk that could prevent/solve the error?
Thanks everybody in the community for the great suggestions. I also want to do my part.
As somebody who does not have access to the repositories I wasn't able to try to fix the issue following the steps that other have suggested in this thread.
What worked for me (and Matillion seems to believe this has no negative impact) was to copy the whole scm folder from the source instance to the target instance manually. Then i remapped the project folder names to the corresponding project ids on the target instance and made sure all files are finally owned by the tomcat user.
For the actual commands to run you can google for the matillion knowledge article about Walk failure. It include some info about where the folder is located, how to find the project ids etc etc. The rest is some basic linux knowledge.
I was experiencing the same problem and after some googling I stumbled upon this post. The problem seems to be that the git history does not get transferred properly on migration.
Doing a "git log" direcly in the source directory on my matillion instance gave the following error:
error: Could not read a2b...
fatal: Failed to traverse parents of commit
78e...
This is resolved with
git fetch --depth=1
git fetch --unshallow
CAUTION: most probably, you will need to execute these commands as another user than the owner of the repo (tomcat). Don't forget to revert all ownership back to the tomcat user with "chown -R tomcat.tomcat /usr/share/tomcat/scm/123". 123 needs te be replaced by the actual number.
After that, using git in the Matillion UI works like a charm.
Hi @MP619204
If you have shell access to the VM you would need to to ask for access to the git repository and set up your private key in the root users .ssh folder.
Apparently, the problem only arises when there are uncommited changes in one of the repositories before migrating. You could ask users to make sure everything is commited and pushed berfore migrating.
I hope that helps.
Best regards,
Philipp
Is that something you figured out on your own or did Matillion support tell you about this.
If they are aware why don't they warn about this?
Thanks for the help anyway. Much appreciated.
Regards,
Mauro
You're welcome!
The part with the unclean git repositories was the assumption of @Bryan (see above) and I was able to confirm that on our Matillion instance. Applying some git trickery was my own idea. I've been working with git and Linux systems for the past couple of years (if not decades) and it seemed quite natural to me to meddle with the repo (knowing that I had a working instance of Matillion ETL as backup).
Just speculating here: Git integration is something that is not used by many users, since Matillion ETL has an internal versioning system. Hence, there are not many reports on bugs.
We have a workflow, where we develop pipelines on the testing environment and pull it to the production environment from our git repository. We're a small team and we can afford making sure that all projects are clean before migrating (from now on, anyways).
Best regards,
Philipp