Follow redirect for API Query component

Matillion ETL
1

Hello,

 

We have an API endpoint that produces a redirect URL that the client needs to follow to get the data. Typically this is done using the -L param on curl. How do we set this up in Matillion?

 

Thanks.

2

Hi @saqib​,

Does the endpoint return an HTTP 302 with the redirect link in a header, or does it return a successful 200 with the redirect link in the response body?

Best regards,

Ian

3

Ian.

 

It returns a 302:

 

HTTP/1.1 302 Found

.....

....

....

Location: https://.....s3.amazonaws.com/..../.../.....

4

Gotcha thanks. I guess it just makes the CYOC fail? It's a pattern that I suspect would get upvoted if you could add it to our Ideas Portal?

 

Meanwhile I think you could make the first call in a script (maybe Bash or Python) just to collect the redirect URL to send to the CYOC component afterwards.

 

Best regards,

Ian

 

5

right. also the wizard for the API Query fails.

 

I wanted to stay away from the python route. But looks like I may just have to do that.

6

Hello @ian.funnell​ . Can you please provide some python code to collect the redirect URL? Thanks.

7

Hi @saqib​,

Here's an example Python 3 script. If "requests" is not installed already you'll need run this command from an SSH session as root to install it.

python3 -m pip install requests

Please see this document for more details about Python module installation.

Then, in a Python3 Matillion component:

import requests

# The variable jv_firstaddr contains the first address

resp = requests.get(f'http://{jv_firstaddr}', allow_redirects=False)

if(resp.status_code == 302):

newaddr = resp.headers['Location']

print(f"Following redirect to {newaddr}")

context.updateVariable('jv_url', newaddr)

# The variable jv_url now contains the redirect address

else:

raise Exception("Was expecting a 302")

I think this editor will remove the formatting from the above, so you'll have to add the spacing again. The script captures the redirect URL from the Location header and saves it into a Matillion variable.

Best regards,

Ian