How can I retrieve a job's component's type using the API?

Hi,

When exporting a job with the API, I can see in the JSON response the list of the components in that job, and for each component, the list of its fields, but I cannot find the type of the component.

 

To explain better with an example:

 

A "Table Input" component is made of the following fields (on Matillion for Redshift):

  • Name
  • Schema
  • Table Name
  • Column Names
  • Trim Columns

... and those fields appear in the JSON response, but there is no information telling that the component is a "Table Input".

 

I could check all the fields and deduce it is a Table Input, but this would be a quite risky way to proceed.

Is there another way or is that information simply missing in the response?

 

Thanks for any help!

Cyrille Dakhlia

 

 

 

Here is an extract of a transformation export:

{

"objects": [

{

"jobObject": {

"JobType": ".TransformationJob",

"id": 282,

"revision": 2,

"created": 1631711226750,

"timestamp": 1631711226750,

"components": {

"288": {

"id": 288,

"inputCardinality": "ZERO",

"outputCardinality": "MANY",

"implementationID": 1354890871,

"x": 892,

"y": 256,

"width": 32,

"height": 32,

"inputConnectorIDs": [],

"outputConnectorIDs": [

299

],

"parameters": {

"1": {

"slot": 1,

"name": "Name",

"elements": {

"1": {

"slot": 1,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "matillion_us_state_lookup"

}

}

}

},

"visible": true

},

"2": {

"slot": 2,

"name": "Table Name",

"elements": {

"1": {

"slot": 1,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "matillion_us_state_lookup"

}

}

}

},

"visible": true

},

"3": {

"slot": 3,

"name": "Column Names",

"elements": {

"1": {

"slot": 1,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "name"

}

}

},

"2": {

"slot": 2,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "code"

}

}

}

},

"visible": true

},

"4": {

"slot": 4,

"name": "Trim Columns",

"elements": {

"1": {

"slot": 1,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "No"

}

}

}

},

"visible": true

},

"5": {

"slot": 5,

"name": "Schema",

"elements": {

"1": {

"slot": 1,

"values": {

"1": {

"slot": 1,

"type": "STRING",

"value": "[Environment Default]"

}

}

}

},

"visible": true

}

},

"exportMappings": {},

"activationStatus": "ENABLED"

},

I apologise for the really bad formatting... and I cannot edit my message :x

It sounds like the Matillion Data Pipeline Language (DPL), described at Matillion's Data Unlocked Conference, is more what you are looking for?

@ian.funnell​ Couldn't exactly understand how DPL will help, or may be co-pilot/auto-documentation, where you can filter on the documented part to find out what all components it uses if it documents it. I was looking for something similar, so we have got environment and need to tune it or for that matter get rid of components deprecated or have better substitutes in terms of performance, now how would you tell which project --> job and as job goes within job itself so more child jobs have that component. Let say we want to get rid of jython and replace it with python, now how do we find which all jobs have jython in it other than manually going in there and finding out which is arduous and pathetic to be honest, I thought the APIs or something similar should give and idea of what components are used in which job, it's not an idea for idea portal its more like must have meta data of the job itself through cli or so.

Hi Cyrille,

 

The REST API is not intended to be used for inspecting jobs, so I don't think there's an easy way to do what you are attempting. The closest to what you are looking for is the implementationID, but it's an opaque reference (like a token). We don't publish a list of what the implementationIDs mean, and they change regularly from release to release.

 

If you can summarise what you're trying to do, I'd say please add an idea onto our Ideas Portal, and we will be able to assess the impact on the REST API.

 

Best regards,

Ian

> The REST API is not intended to be used for inspecting jobs, ...

 

Why?

 

Could you please explan?

 

And this is not only via the API, but also via exporting jobs via the GUI.