I am using Python script to update the variable but I realized that the updated value is only available for the current job run and the value goes back to its default state once the job finishes.
Unfortunately Matillion doesn't save the state of a variable. The best bet here could be to use Environment variables, and override everytime when the job is run. Or the best option would be to use the underlying CDP to save the value in a table. And update/retrieve it for next runs.