Can we create a shared job for accessing multiple API sources?

Matillion ETL
1

Hi,

 

I am new to Matillion and I am working on creating shared jobs that could be used for ingesting data from multiple sources.

 

We have created a shared job that could ingest data from file shared on different data cloud.

 

Similarly, we are attempting to create a job that could extract data from multiple APIs which are shared by different source systems.

 

Has anybody attempted this earlier? Any inputs will be helpful.

 

Thanks

 

 

2

Hello @User16387910083285550412​

This sounds brilliant, have you had a look at the Matillion Exchange? many users have created and uploaded shared jobs there for fellow users to download and use with their Matillion ETL setup alternatively if the shared job you desire is not there, please raise it as an idea on our ideas portal.

Kind regards, Joe

3

Hi @User16387910083285550412​,

I have not created a shared job to do this but I have done this using 2 different methods. The first being Python which gives you tons of flexibility but requires you to write the code. The second is by making multiple API Query calls in parallel and the bringing the results together using a transformation job or Python script. It can get hairy but definitely doable.

The path you choose completely depends on what you are trying to do, the API's you are dealing with, and your skill level in Python.