Am I crazy or does this not work... I'm trying to output an array from an api to a grid variable. It runs but I don't see a thing in the grid variables selection... So I go to the basics and do what is in the documentation. https://documentation.matillion.com/docs/2917841
array = context.getGridVariable('people')
for data in array:
print(data)
context.updateGridVariable('names', array)
I have both grid variables set up, with the 'people' grid variable having 5 generic peoples names ['matt', 'frank', 'glen']. Then I run the above code, and I don't see an output to the 'names' grid variable. Is that intended? My expectation is that I will see the names in the 'people' grid variable in the 'names' variable.
Just to clarify. It sounds like you have a Grid Variable call people. Within that grid variable you have a column (names) defined. In that grid variable you have 5 rows with matt, frank, and glen. If this is correct then I can see a couple issues with the code you posted.
Let's assume your grid variable looks like this: (Note: The best practice would be to start your grid variable names with gv_. As an example you would normally use something like gv_people as your Grid Variable name)
The one thing I would point out about your line of code is that it looks like you might be trying to use the column name (names) within the grid variable instead of the grid variable. You can think of a grid variable as an array with nested arrays. You are not updating portions of the array when you call the updateGridVariable. You are updating the entire Grid Variable. This means you are not updating a single column you are updating the entire Grid Variable. This means if you have 2 or more columns then all the values need to be defined in the array when you execute the updateGridVariable. Hopefully this helps. Please let us know if you need more guidance. Thanks for posting!
Really appreciate the feedback. Super helpful. I still am wondering if I'm supposed to see any updates that I make in a grid variable?
For instance, if I add another name to the people variable using - context.updateGridVariable('people', [['Kyle']]) I can see the new name added to the variable via print(context.getGridVariable('people')), however, when I go to the grid variable wizard Eric is not there. On top of that, I created another python component and used print(context.getGridVariable('people')), but the name Kyle that I'd added earlier was not in the list.
I see what your asking... So, lets start with how Grid Variables work. At design-time, you first create the Grid Variable using the UI, the values you add are just default values. Those default values exist when you first start the job. At run-time, if you do something like updateGridVariable, whatever you updated the Grid Variable with is what it will have until job finishes. As soon as the job finishes, the Grid Variable is exactly what it was before the job started. Meaning it reverts back to it's defaults. This is the difference between run time and design time.
The idea is that for each job run, the values in the Grid Variable could dynamically change based on that particular job run. As an example, you could make an API call and save the results to a Grid Variable that had 1 or more columns. From there you could process each result individually. The kicker here is that with each job run the API call may return a difference result set each time.
In your example, if you created a Grid Variable called people and the default values were ['matt', 'frank', 'glen'], you then run the job the job or just the component that executes context.updateGridVariable('people', [['Kyle']]), your new value will be just Kyle until the job ends and then it will revert back to the default ['matt', 'frank', 'glen']. Keep in mind when you run context.updateGridVariable('people', [['Kyle']]) you are replacing the default values with just Kyle not adding to what is already existing. Therefore your Grid Variable for that run will only contain Kyle. If you want to add to it then you will need to get the Grid Variable values and then add on to those values and then update the grid variable with the whole collection which would include the Kyle you added. Hopefully this helps clarify this a bit more.
One more question, and this is just for qualification.... Plus I could just test, but when you say job, you mean the whole orchestration? For instance, if I were to add another component after the py script, it would recognize the new fields added to the grid variable, until the job finished.
You are correct (whole orchestration run = job). If your GV has a single column called Names and it defaults with 1 name. You can execute the job and the first component is a Python script which replaces the 1 name with 10 names and then updates the GV. The next component will recognize the 10 names and can be used. (Typically you would iterate over the names using the Grid Iterator.) As soon as that orchestration/job finishes, the GV reverts back to default.
What is happening under the hood is that when the orchestration/job starts Matillion is creating a variable with the array of names in memory on the server. As you update the GV with new names it's redefining the in-memory variable with new values. As soon as the orchestration/job finishes the used memory for holding those values is cleared and released back to the system. The next time the job starts again the same process starts all over. I hope this helps.
In a Matillion Python Script component, grid variables are represented as a list of lists. So it sounds like you might be trying to update a grid variable using a scalar value (string or int) instead of a list?
E.g.
context.updateGridVariable('gv', 1)
Results in: java.lang.Integer cannot be cast to java.util.List
And
context.updateGridVariable('gv', 'x')
Results in: java.lang.String cannot be cast to java.util.List
This thread should be starred and linked to the Grid documentation. It is really helpful.
If anyone is still listening, can I add/remove columns from a grid in python? I need to dynamically create a grid based on api response and the write that grid to a table!
