File Iterator for https url?

Matillion ETL
1

Hi,

 

I need some inputs on below task.

 

1) We need to hit https server (which has a userid password and post login, shows list of .pgp files which are encrypted)

 

2) Download all these available files into S3 bucket (we tried to use file iterator component, but looks like it doesn't support https source.. We are able to download one file at a time and decrypt using step 3)

 

3) Decrypt these files in S3 bucket and store as .csv (we are able to do this via Python component. But we manually uploaded multiple files into s3 bucket, as we were not able to download multiple files using file iterator component of Matillion )

 

4) Load all these decrypted csv files into one snowflake table

 

 

Is there any way using which we can perform step 2) ?

 

Any help on this will be appreciated. I have already referred below two threads , which seems helpful, but issue is that my source is "https", not sftp or ftp. Hence, I am unable to iterate via file iterator component. Looks like I need to use first "Gird iterator" along with "Append to grid" and then use "file iterator" along with "Data Transfer" component.

 

https://metlcommunity.matillion.com/s/topic/0TO4G000001bpFLWAY/sftp-server

 

https://matillioncommunity.discourse.group/t/is-it-possible-to-transfer-files-using-a-file-pattern-in-data-transfer-component-or-any-other-component/1167

 

Thank you!

 

2

Hi dbkulkarni,

 

For an iterator to work you would need to obtain a list of the files that the server has.

 

But normally an HTTPS server goes to considerable lengths to prevent anyone obtaining a list of the files it has.

 

Is there any way to get that list from your server? That would determine how to build an iterator.

 

Best regards,

Ian

 

3

0694G00000EupUWQAZ_0D74G000007jAVlSAM
0694G00000EupUWQAZ_0D74G000007jAVlSAM1136×411 123 KB

Here is the error we are getting when we use file iterator against this https protocol / url

4

@ian.funnell​ - Considering we are able to get the list of https server files, how can we build this iterator? Is there any working example which I can refer to?

5

Interesting... It's unusual for an HTTPS server to permit that.

 

How do you get that list of https server files?

6

Trying via Python script.

​Is there any way to use these file names to iteratively download from this server?