Delete or move files within sftp server

Matillion ETL
1

 

I transform the csv files received in my sftp folder to s3 using the job defined. further, the files are copied to an archive folder in the sftp server using the data transfer component. (the files are copied, as I could not find a way to move them)

 

 

 

my queries are

 

  1. Is there a way to move the files within a sftp server between the folders?
  2. Or, how can I delete the files from the sftp folder?

 

 

a solution to any one of the above queries would solve my problem.

2

Hi @jubith.kariyath​

you can support this idea to add support for a "move" operation to the Data Transfer Component. Currently it is not possible to actually move or delete files with Matillion's native components.

You will have to write a Python or Batch script using the AWS API/CLI to handle your file operations.

3

Thanks for the heads-up about supporting that idea Michael. Ridiculous that a modern ETL doesn't have something as simple as file archival (moving files!) built into the components...

4

I ended up writing my own job that I've shared to the Matillion Exchange. This is for S3 file movement only however.

 

https://exchange.matillion.com/s/exchange-job/a074G00001MN17BQAT/s3-move-file

5

I agree. It does not make sense that I must run the delete operation outside of my ETL solution.