# How do you loop an entire pipeline to extract multiple csv files in batches?

**URL:** <https://matillioncommunity.discourse.group/t/how-do-you-loop-an-entire-pipeline-to-extract-multiple-csv-files-in-batches/2345>\
**Category:** Matillion ETL\
**Created:** [December 1, 2023, 1:59pm UTC](https://matillioncommunity.discourse.group/t/how-do-you-loop-an-entire-pipeline-to-extract-multiple-csv-files-in-batches/2345 "2023-12-01T13:59:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![matillearner](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@matillearner](https://matillioncommunity.discourse.group/u/matillearner)\
**Post date:** [December 1, 2023, 1:59pm UTC](https://matillioncommunity.discourse.group/t/how-do-you-loop-an-entire-pipeline-to-extract-multiple-csv-files-in-batches/2345/1 "2023-12-01T13:59:57Z")

</div>

I am quite new to Matillion and am struggling to create a pipeline that can loop through a table to extract data in batches. Each batch should then uploaded from the S3 bucket to a SFTP folder as a separate csv file with a future date stamp.

&nbsp;

Example: Extract 100 000 records in batches of 25 000. Each batch is exported in a separate csv file with a date stamp. The first file date is today's date, the second file is dated tomorrow, the third file is dated the day after tomorrow etc.

&nbsp;

&nbsp;The pipe components are as follows:

&nbsp;

Query Result To Scalar (to count records) \> If (to check if any of them are valid) \> Create Table (create a temp table for the specific batch) \> SQL Script (insert the batch to the temp table and to set the flag on the extracted batch) \> Query Result To Scalar (to determine the date stamp of the file) \> S3 Unload (pull the data from the temp table into the csv file) \> Data Transfer (push the file to the SFTP).

&nbsp;

&nbsp;

&nbsp;

Currently, there is a pipeline for each csv file (more than 10 of them). This delivers the desired results but I believe there must be a much better way to accomplish this than the "work around" of 1 pipe per file.

&nbsp;

Is there any way that I can run all 10 batches through just the one pipeline, such as extract the first batch and upload, extract the second batch and upload, until all files have been created?

---

<div class="post-metadata">

**Author:** ![ASD](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/asd/32/1323_2.png) [@ASD](https://matillioncommunity.discourse.group/u/ASD)\
**Post date:** [December 8, 2023, 5:26pm UTC](https://matillioncommunity.discourse.group/t/how-do-you-loop-an-entire-pipeline-to-extract-multiple-csv-files-in-batches/2345/2 "2023-12-08T17:26:08Z")

</div>

Have you tried using File Iterator?

[https://docs.matillion.com/data-productivity-cloud/designer/docs/file-iterator/](https://docs.matillion.com/data-productivity-cloud/designer/docs/file-iterator/)