Hey folks!
I have recently started using the Matillion Data Productivity Cloud, and while I’m loving its flexibility, I’m curious about how I can get the most out of it in terms of efficiency. My current data workflows are a bit slower than I’d like, especially with larger datasets. When I was Googling for the same I came across these resources/articles https://www.matillion.com/blog/matillion-listens-to-user-feedback What is Mulesoft however they are good but I want to learn more from community members.
Does anyone have any suggestions for optimizing performance on the Data Productivity Cloud? Whether it’s for speeding up data loads, transformations, or general productivity tips, I’d love to hear about your experiences and any hidden features you think are worth exploring!
Thanks in advance.
Hi @leenasmitth
Thank you for your post, I had held off to see what other users wanted to share with you, they may are being shy but I would love to certainly find out how you have found the experience of using the Data Productivity Cloud and if there are any areas specifically we could help and get you in contact with the best people to support in this :)
Kind regards, Joe
Hi Leena,
You have lots of options but it will really depend on what you are doing.
- Data Loads: Usually the bottleneck is the API provider. CSV and SQL loads are usually very fast. APIs not so fast, so you can use features like "expand" that some APIs offer. I reduced a run from 2 hours to 2 minutes using this feature. Also parallel running may help in this case. In any case avoid using iterators, they are slow, as you are performing a lot of calls. Same for SFTP and the like, the iterator creates a connection each time, therefore it is pretty slow to, for example, list files. Use Python instead.
- Transformations: It will mainly depend on your database. For example Snowflake allows you to increase your computing capability, speeding things up significantly. In any case a good Database/Query design is the base for running transformation efficiently.
- Hidden features: Recently I requested Matillion to implement a more flexible pagination system for APIs, and they actually developed a very powerful pagination system! Initially it may feel complex but it is quite straight forward if you are familiar with APIs and/or coding. Also they have offered excellent support to use it. Another one not so hidden feature is the new AI component. You can transform data using AI. I have not used it myself yet but I have seen a demo and it is truly impressive!
If you share your use cases probably myself or other users may be able to give you more specific tips.
Good luck with Matillion DPC!