Hi - I am using the Data Transfer component to download a single SFTP file. The file name changes daily and includes a date stamp (yyyymmdd), e.g. 'test_file_20210905.csv'. How should I set the file path parameter in the component to look for this specific file?
Hi @Singularity,
I would say there are 3 parts to this:
- Create a Job Variable in your Orchestration Job
- Update the variable with the date stamp - use a Python Script component
- Copy the file - use a Data Transfer component
I did a quick test using my Matillion server itself as the SFTP source, and ended up with this...
I named my variable today_yyyymmdd and chose the Text datatype
In my Python Script:
from datetime import datetime
context.updateVariable('today_yyyymmdd', datetime.strftime(datetime.now(), '%Y-%m-%d'))
Then in my Data Transfer:
Note I deliberately chose different names for the source and target:
- Source name: /tmp/${today_yyyymmdd}.csv
- Target name: copied-${today_yyyymmdd}.csv (inside the bucket which I have pixelated)
Best regards,
Ian
Thanks @ian.funnell I will give that a try. I'm a new customer and loving the platform so far - just to confirm, the ${job_variable_name} syntax can be used in any type of component parameter field?
Also, in the Python script, is any particular library import needed to access the "context.updateVariable" function?
That's correct @Singularity the variable syntax is ${<variable name>} which you can use widely. More information in this article.
In a Python Script, no special import is needed to get the context object: it's an implementation specific to Matillion ETL. Note that in a Python Script the syntax ${variable} is not required, you may simply use variable. More information in this article.
Best regards,
Ian
Hi @ian.funnell thanks again for the feedback on this. I was able to do this and it appears to be working but a clarification on your original post is that the context.updateVariable() function appears to only work on Environment Variables and not on Job Variables.
Also, does the re-set Environment Variable persist outside of the run time of the Python script in the job, or is it only during the runtime of the job? In the Manage Variables UI I only see the 'default value' of the Environment Variable, and not its currently set value after my Python script runs, as I would expect. Does the variable revert to this 'default value' after the job completes?
Hi @Singularity, good to hear you have it working!
Strange, but, for me it seems to work fine with Job Variables too, e.g...
Then
Regarding your second point, you're exactly right: variables revert to their default values after the job completes. Please see this document for more detail on variable lifecycle.
If you want to update the default value of a variable, then you need to use the REST API. For example I have a Text one named today_yyyymmdd containing "today's date" in a well defined format. It obviously needs to be updated every day so I have a job which runs at 0h30m to update the default value. The job has a single Bash Script which contains just this curl command:
curl -u "myusername:mypassword" -X POST -H "Content-Type: application/json" "http://localhost:8080/rest/v1/group/name/${project_group_name// /%20}/project/name/${project_name// /%20}/environment/name/${environment_name// /%20}/variable/name/today_yyyymmdd/set/value/$(date '+%Y%m%d')"
Unfortunately you can't use saved passwords in a Bash Script so you have to find a different way to present them to the API call.
Best regards,
Ian
I have this similar requirement but not with SFTP, I need to move the file from azure blob storage.I have tried to do the same but when i pass the blob location as the variable, it throws error:
Please suggest
Thanks @ian.funnell I tried on a test script and it worked like your example, so I will update my job and see if I encounter the issue again - not sure why it wasn't working before, probably user error. I was initially thrown off further testing of the Job Variable since the documentation for the context.updateVariable function only references Environment Variables and not Job Variables so I assumed that was the intended behavior. See screenshot.
Also, is the ability to update the default values via the API, as you describe, a new feature since the documentation on that describes that the default values can only be modified manually and not via API. See second screenshot.
