Ask Matillion Anything w/ Director of Product Matt Farmer — Data Productivity Cloud | July 10-14

Data Productivity Cloud
1

Hello everyone,

 

We are thrilled to announce the launch of our latest series of interactive Q&A sessions with members of the Matillion team.

 

Our first guest is Matt Farmer who is the Director of Product at Matillion. Matt has been working hard to deliver the Matillion Data Productivity Cloud which launched Tuesday, June 27.

 

0694G00000SxBB2QAN_0D54G00009RdKHPSA3
0694G00000SxBB2QAN_0D54G00009RdKHPSA31200×628 96.8 KB

 

We know you will have many questions about the Data Productivity Cloud ranging from functionality and how it will interact with Matillion ETL/MDL.

 

Please reply below with any questions you’d like answered and Matt will respond during the week of July 10!

 

Cheers,

Chika

 

 

2

Do you envisage DPC being used as an integral product for engineers who work with AI language models?

3

Fantastic, I do have a few questions.

 

  1. Will DPC provide any options for integrating with a specific corporate Git, such as the Matillion ETL?
  2. Is it planned to add grid variables to the DPC?
  3. Is there a direct way/tool for helping migrations from Matillion ETL to DPC?
  4. Is DPC going to have shared jobs? If so, will Shared Jobs created in Matillion ETL work on DPC?
  5. Is Python on the DPC roadmap? If so, will the instance's compute processing costs vary by utilization, or is the instance size fixed, with no effect on the ultimate price due to consuming additional processing power? 

 

Sorry, I like to make questions 😂

4

Will DPC

 

  1. have SoX Compliance Security Model with RBAC and AUDIT of each Object?
  2. Does it support Multiple Environments like Dev ,QA? Test & PROD
  3. Does it have Reporting on Jobs -how they perform over time
  4. Does it have LDAP / AD integration for users and Groups for SSO , Auth and Access
  5. Does it perform Well with large compute
  6. Does it auto Scale based on Job/Record volumes?
  7. Does it support Git CICD
  8. does it has APi for each action
  9. Does it have Variables
  10. Can it be easily migrated from Matillion ELT/ETl to DPC

 

5

Hi Martin - great question! I think the short answer is yes - absolutely. Essentially preparing data for a Large Language Model or other AI workload will still require bringing in data from multiple sources and preparing it to be productive. This is at the core of what the Data Productivity Cloud is designed to do.

 

There are going to be additional features that will bring more value to this use case and we are planning several. However, I'd also be very interested to hear what you would like to see in this space?

6

Hi Marcus! I'll tackle your questions in order:

 

  1. Yes, we will very soon be providing the ability to connect your own GitHub account to Data Productivity Cloud. All your files, commits etc will be stored in your own GitHub source control. We have aimed to do this in a very 'Git neutral' way - it should work as you would expect Git to work. We will be adding support for other Git providers too - please let us know which ones you would like to see!
  2. Yes Grid Variables are currently being worked on, you should see support for them soon.
  3. We are not currently recommending migration from Matillion ETL to Data Productivity - existing customers should contact their Matillion account team or register their interest in migration here: https://pages.matillion.com/customer-migration.html. Will we be producing tooling and guidance? Absolutely - please watch this space!
  4. Yes -we are currently working on features that bring the value of shared jobs to the Data Productivity Cloud - it's been a top request.
  5. Python is available today with the Hybrid agent. We are working to bring this to the fully managed Agent in the new future. Python processing on the agent works in a similar way to Matillion ETL, i.e. it is designed for smaller workloads running in a single task. We are looking at ways to support larger Python workloads, so please let us know if this would be a feature you are interested in?
7

Hi Ranzith - many thanks for your questions!

 

  1. RBAC support is available now, we have more to come in this area soon with the addition of custom roles and more objects. Audit is also a feature on the near-term roadmap.
  2. Yes - additional environments can be added to support this workflow. We plan to add more tooling & guidance in this area to help customers set up common deployment scenarios.
  3. All pipeline runs are reported in the dashboard. How they perform over time is a great suggeation, and something we are looking at on the roadmap.
  4. This feature is coming soon.
  5. Our PipelineOS execution engine has unlimited scalability as one of our core design principals. So Yes! If you have a particular scenario you'd like to investigate, please reach out to your account team.
  6. Auto scale is not available today, but is a planned feature on the roadmap.
  7. Yes! We will very shortly be adding support for attaching your own Git provider, where you can use all the power of that service to manage CI/CD etc.
  8. We plan to offer a full suite of APIs and a great API developer experience soon.
  9. Yes, variables are available for each pipeline. Grid variables are currently being worked on.so expect to see support soon.
  10. We are not currently recommending migration from Matillion ETL to Data Productivity - existing customers should contact their Matillion account team or register their interest in migration here: https://pages.matillion.com/customer-migration.html. Will we be producing tooling and guidance? Absolutely - please watch this space!