Updaten grid variable (null)

[m.w]

Hello

we try to ingest data from SQL SERVER into Snowflake

​

therefor we are using the JDBC incremental compoenent.

​

We though we can script the TABLES; COLUMNS;IS_INCREMENT,IS_KEY

​

1. so first step is defining all the tables we want to ingest => define in a grid variabel (ok)
2. reading all column meta data for all tables from source database (sql server) (ok)
3. having a pyhton script which generates a grid variable with the table, column, is_increment, is_key which would be used in the "jdbc incremental component" (update the grid variable is causing some trouble)

​

here is the script

 

``py_my_grid_variable = context.getGridVariable('gv_tables_cols')

print(py_my_grid_variable)

​

# Only do the rest if the grid variable is not empty, and contains more than 0 values.

# Otherwise the default, empty value of py_my_scalar_variable will persist

gv = []

py_my_scalar_variable = ''

if py_my_grid_variable is not None:

 if len(py_my_grid_variable) > 0 :

   

  ## Matillion grid variables are lists of lists. 

 

  for x in py_my_grid_variable :

   g = []

​

   if x[0] == jv_column_isincrement :

    is_increment=1

    is_key=0

   else :

    is_increment=0

    is_key=0

   

   g.append(jv_table)

   col = str(x[0])

   g.append(col)

   g.append(is_increment)

   g.append(is_key)

    

   gv.append(g)

   

  a= context.updateGridVariable('gv_metadata', gv)

  print(len(gv))

  print (context.getGridVariable('gv_metadata'))

   ``​

 

the problem is that   

 

a= context.updateGridVariable('gv_metadata', gv)

print (context.getGridVariable('gv_metadata'))

 

it is updating the gridvariable a kind of but is failing for some reason

when we print out the grid variable gv_metadata than there is some null after the array

which is strange , because when we print the array of list gv than it looks fine

 

Does anyone have some ideas why the context.getGridVariable('gv_metadata')) is not properly working?

 

Thx Martin

​

[m.w]

whole output looks like

[['property_id'], ['DS_Load_ID']]

2

[['properties', 'property_id', 0, 0], ['properties', 'DS_Load_ID', 1, 0]]null

 

if I print the update grid variable at the end it seems that it has

[['properties', 'property_id', 0, 0], ['properties', 'DS_Load_ID', 1, 0]]

put for some reason the updategridvariable makes some null at the end

and the component is then failing without any further message

[m.w]

so the pyhton component is failing without any further information

and if I than say in case of an failure use a next python script to print the grid variable

the variable is update

 

put I dont know why the python script reports a failed task

[m.w]

here you see the workflow

and you see that the first python script is failing