Hello,
I would like to know if there are customers that already started using Matillion for Databricks and if so, what are your experiences so far? I'm particularly interested in performance when building and executing jobs, and some limitations you might found.
We started with POCs on Matillion for Databricks, and so far we are experiencing longer job executions and component/job validations comparing if we would submit the same work directly on Databricks. We got recommendations from Databricks to use sql warehouse instead all-purpose clusters, even though Matillion supports both. So we doubled size of SQL warehouse and used unity Catalog, and that helped a bit to run Matillion jobs faster. Volume of data was in range of ~2 mil records, which is really not much so we were surprised that simple select would take some time (~ 20s). Matillion and target Databricks are in same region, so we excluded network latency.
Also, in terms of running parallel jobs, i know there is limit of 16 jobs on Matillion side, but would there be some bottleneck on Databricks side?
Regards,
Tanja