Moved the data from S3 to SFTP server in my Matillion job. Is there a way to automatically move the files on SFTP to the current folder then after certain number of days - to the archive folder and then purge the files after a while?
Hi @OI138049, you could potentially setup and orchestration to handle this. You would have to schedule the orchestration to run once a day (ever day). The moving of files to the current folder should be straightforward. The archiving of files after a certain number days would take a bit of setup. What I would do is get the list of files that get moved to the current folder and wrote those filenames, the date they were moved to table. Then when the job runs each day it would check the table and determine if any files need to be archived, or purged based on the moved date. Depending on the threshold you set for archive and purge you would take the appropriate separate action based on those conditions.
There are probably better tools out there to do this kind of operation but it's definitely possible to use Matillion to accomplish it. I hope this helps give you some ideas.