Our SDLC involves persisting the version of shared jobs such that manual reconfiguration of dependent jobs is not necessary when new functionality is added to the shared jobs. However, we've discovered that upon deployment of an updates shared job (again, to the same/existing version), the parameters for the job seem to get shifted or re-ordered, which means they get out-of-sync with the values specified for them.
The effect of this is that we can't simply create and deploy an updated shared job...we still have to go in to all dependent jobs and update/confirm the parameters.
Is there any way to get the parameters to be consistent? Thanks in-advance. -Ben
Parameter binding into Shared Jobs is by position, not by name. So you do need to be careful to keep the columns in the same order whenever you make a new revision of a Shared Job. The Copy From Existing option can be helpful in this respect.
Is it possible your parameter ordering may sometimes be changing between revisions?
Hi Ian. Thanks for your swift response. I'm still struggling. The Copy From Existing does not appear to be documented well - there is a dearth of information about what it actually does, and how to effectively use it. I did try it, but I could not discern its effects.
When presented with the list of parameters in the Shared Job creation screen, I carefully ordered them per the screenshot I have of the parameters as listed for the job in-use in another job. However, this didn't work either - still many mismatches.
I admit to being quite baffled as to this design concept: why on earth would the position of a named parameter be important? It clearly causes significant confusion and problems, and in no way builds integrity into the solution.
I'll keep pounding away at it - perhaps the logic will be revealed. Any further counsel you (or anyone) can offer would be super-helpful. Thanks again! -Ben
Hi @Benjamin, you will definitely want to use Copy From Existing if you are replacing a specific version of a Shared Job. It will not only save you a ton of time but ensure your parameters are in the right order. What is happening under the hood when you use Copy from Existing is that it's basically taking the configuration from an existing shared job and version and applying it to the new one you are creating. It's universal in that you can use it to overwrite an existing version of a Shared Job or for creating a new version of an existing Shared Job or creating a new Shared Job all together. You can think of Copy from Existing being a copy of everything from another shared job until you get to the actual orchestrations or transformations that make it up.
I would agree with you based on the underlying job variables within the existing version of the Shared Job and the new one that will replace the existing version Matillion should know which parameters to map to. If you posted that to the ideas portal, I would vote it up. I know that the functionality and implementation of shared jobs within the Matillion product has been a topic of conversation in the past. I think it serves a basic purpose but could be much better in areas like this one.
Hi Bryan - thanks so much for providing some additional details on this topic. I'll explore Copy From Existing a little more thoroughly (not sure why it didn't seem to be effective on my initial passes with it).
And, I'll get an idea posted up when I get a breather from a breakneck week - and will tag you.