My table has 72,000 results for each one I have an action that needs to be done. This is a historical re-sync job. Table iterator has an odd limit of 5000 iterations. Is there any way around this ?
I mean I don't really see usefulness of the iterator component if it's limited to 5000 in world of big data.
Use Bucket if no of iteration is more than 5000 to generate a list buckets of 5000 per iteration and once 5000 is processed invoke a parent orchestration job which will in turn invoke the the orchestration job having the table iterator / grid iterator. Self recursion is not supported in Matillion. Parent Job would be required for recursion logic.
Custom logic implementation is required but you can have good control over the pipeline. I have done this it works like magic
Solved by using table pagination.
What do you mean by buckets ?
Bucket means divide 72k records for each 5k list and loop for 15 times using a parent orchestration job