Hi,
I have come to the conclusion that the best course of action is for us to have 2 separate copies of Matillion ETL for BigQuery (medium) running in two GCP projects - one for Dev, and one for Prod. The Dev one will be on all day, but the Prod one would only be on when it needed to run schedules.
Doing this would allow me to develop on one environment, then package that up and deploy it over to production with confidence.
Has anyone else taken this approach?
- Rob
There are several ways to organize your Matillion environments. Your approach sounds perfectly reasonable if that matches your requirements.
Our team operates one "Production" instance which hosts one "Development" project and one "Production" project. This instance is running during the nightly batches and during office hours if active development is happening. The PROD project always points to the Production database in our Snowflake account while the DEV project is connected to the Development database on the Snowflake side.
Similar to your suggestion, we also develop on the DEV project and once we are confident about quality and stability of our processes, there is a deployment process to the PROD project.
We also own a second Matillion instance for testing Matillion Software updates or special tasks in a seprarated environment.
I think the "best" solution depends on your team size, performance and security goals, and the available budget. As mentioned above, your approach is absolutely fine!
Thanks Michael, thats really helpful, more helpful than the reply I got from Matillion Support on the matter! I wanted to be sure I wasn't just trying to do things "the old way" and it still made sense.
I don't know if its the same for your platform, but on GCP we've got the choice to launch a new instance from the marketplace into the second project (which is what we did originally for the first one) or I believe there is a process to copy the existing VM over to the second project... I'm not sure which would be the best option.
We usually spin up a fresh instance with the Marketplace image and then use the "Migrate" feature which is built into Matillion ETL.