I know that this issue has been discussed in the forum and there are some ideas and enhancement suggestions for it. In the meantime, I developed something that works but I am not sure if it's a really bad idea.
I am trying to execute multiple jobs in sequence. The jobs belong to different projects. I would also like the 1st job to complete, before the next one starts. The solution that I found to execute a job from a different project is to call the proper Matillion API with project/job/version/name/environment from Bash. I capture the JSON that is being returned and retrieve the Execution ID.
Subsequently, I check for the status of the job, until SUCCESS or FAILED is returned. I then loop (in Bash again) every 10 minutes - or whatever time is appropriate - and check again. The loop looks something like this:
for run in {1..10}; do
JOBSTATUS=$(curl -k -X GET -u user:pass
http://localhost:8080/rest/v1/task/id/$P_JOBID -s | jq -r ".state")
if [ $JOBSTATUS = 'SUCCESS' || $JOBSTATUS = 'FAILED' ];
then
break
else
sleep 3600
fi
done
echo $JOBSTATUS
So, this job is basically running, possibly, for hours until it is ready to exit and execute the next job.
My question is: does this put a tremendous load on the Matillion server? What if Matillion is running on a single processor? Will that basically slow down all other executions/jobs that might be running in parallel? And would implementing an enhanced Retry Component make a difference in the load to the Matillion server?
Thank you,
Aristotle
