Grid Iterator generating more iterations than my SQL query

Matillion ETL
1

Hi there,

I am on METL 1.68.26. We are migrating to DPC soon.

Just experiencing an oddity - am using a Query Result to Grid component to capture a set limit of 100 rows:

image
image256×136 4.37 KB

When running the SQL I get 100 rows as expected.

This component is hooked to a Grid Iterator.

image

When running the pipeline the iterator is generating more than 100 rows:

image
image267×194 7.22 KB

This example it generated 800 iterations, but sometimes it will be 400 sometimes 500! :smiley:

It seems to work fine just very curious why it appears to be creating more work for itself? Is this Matillion handling ‘do-overs’ automagically or is the log wrong?

Just wondering if anyone else has experienced this?

2

Hi @SB699925 and thanks for the interesting post!

I took this to the team to see if any of them had any ideas. One of the team has asked if there’s some sort of JOIN in your SQL that is causing the component to generate duplicate rows on the JOIN condition?

Many thanks,
Claire

3

Thanks Claire!

I’ve checked my SQL and run it separately. Anything is possible I suppose but when I run it directly in Snowflake I get the expected 100 rows (the limit condition) and no duplicates.
The Query Grid is returning 800 rows:

image
image1339×213 29.8 KB

Here is the definition of my grid variable:

image
image724×519 16.4 KB

And here are its params:

image
image829×216 16 KB

It seems fairly straight-forward to me, the SQL returns 100 rows and populates the grid variable with the results.

Could it have something to do with the fact that a grid variable is an “array of arrays”?

I set the limit in my query to 5 and it returned 40 rows in the grid. When I run it in Snowflake I get 5 rows. It seems the SQL is interpreted slightly differently in Matillion than directly in Snowflake and might be applying the limit in a different way.

Thanks!
Steve