I would like to send an api message to Fivetran to suspend a connector replication from a DPC orchestration job just before we do transformations in Matillion DPC. They (FIvetran) have a 'patch' API call that I was able to trigger using Postman (curl) to cause Fivetran to pause a connector. I would like to do the same thing from Matillion DPC (full SaaS).
Thanks
Hey there,
Welcome to the Matillion Community! I (believe) we spoke about this on Reddit, and I was waiting to get this idea made public, which has now happened. https://roadmap.matillion.com/p/support-additional-http-verbs-in-custom-connector I think it needs some additional love! It could use your upvote and a comment!
Thanks,
Johnathan
I've been thinking about this over the weekend, as I appreciate the idea will take some time to come to fruition. If this is Snowflake, you can likely do what you're trying using the Python Pushdown component and a bit of Python to make the API call.
Hi @tnickason (Customer) ,
As John said the only way to do this to code something in Python. You can send WebHook Posts from Matillion but not generic API request. You can still use a Custom Connector if you API requires a GET or POST message but not if it requires a PUT message or any other.
To code the Python you can use AI. I found it good enough, although you quite often need to doble check and correct the code.
In Snowflake you would need to create an "Integration". Matillion has a doc on that: https://docs.matillion.com/data-productivity-cloud/designer/docs/python-pushdown/#snowflake-configuration
Basically Snowflake is completely closed to the external world so if you want to send data out (or in) you need to create a network rule to open Snowflake firewall.
I hope it help
Fernando Hidalgo
Hi Johnathon.
I've have looked into that. We do use Snowflake but we do not have deep knowledge of using Python and/or Snowpark (which I think is required). As time permits I will investigate but going down this road may lead to more questions than it is worth at the moment. Are there any video's describing/showing how to use this component with Snowflake?
Hi Fernando.
Thanks. I've had a quick look at the python pushdown but will need to review more. Not sure what you mean by an 'integration' in Snowflake. We already use a dedicated Snowflake service account for Matillion and created a network rule to allow DPC to connect. Is there another integration type that is required for Python pushdown?
Thanks
Yes, you need to create a network rule to allows Snowlflake to connect to FiveTran. Then add this network rule to a Snowflake Integration and finally add that Integration to the Python Pushdown component.
You do this just once with an ACCOUNTADMIN:
CREATE OR REPLACE NETWORK RULE fivetran_access_rule
MODE = EGRESS
TYPE = HOST_PORT
VALUE_LIST = ('Here your host or IP where you need to send the API message.');
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION gkc_w3schools_access_integration
ALLOWED_NETWORK_RULES = (fivetran_access_rule)
ENABLED = TRUE;
I hope it helps!
