File Iterator doesn't stop when it should

Matillion ETL
1

Hi,

 

My problem is that when using the file iterator to append the file name onto my rows, the iterator won't actually stop when there is no more information in one file, and instead move onto the next file and continue to append the first filename onto this second file, which is a process that repeats for the next 20+ files and is obviously wrong. I've followed this guide to help me do this but no matter what I do I can't get the iterator to stop when there is no more information left in the first file.

 

Hopefully I've explained it well enough in the first section, but just in case, here's an example that is similar to the problem that I'm facing:

 

There are two files. The first file has 20 rows of information in it and the second file has 30 rows. Instead of only having 20 rows of information with the first filename appended to it, I have 50 rows with the first filename appended on, followed by another 50 rows with the second filename appended onto it. How can I get it so that I only have 50 rows in total, with 20 rows for the first filename and 30 for the second?

 

Thanks!

 

2

You might need to pass the file name as a variable which then automatically changes to the second filename when reading /working on the second file. ​

3

Hi @zachary.ryan​,

Pardon the delayed response. Did some digging and there are no open tickets for file iterator so I presume this has something to do with the way you have configured the job?

It would help to review the job design to be certain what the issue is. You could create a test job to replicate the issue and share it here on the Community so we can get to the bottom of this. You can also try @AnudeepK​'s (thank you!) approach.

Hope to hear back from you, @zachary.ryan​

Cheers,

Chika