I am working on matillion for creating data pipelines, and it would be great if I can get my following queries sorted out:
- While ingesting limited number of columns from any csv file that is stored in Azure blob storage, I am unable to get the exact output.
For example: If I have a csv file dataset of 10 columns. I just need to fetch 4 columns pout of it.
In this case, I am unable to fetch the required limited number of columns. Either the output is null or in some cases, the matillion was fetching the first n number of columns (if the required number of columns=n).
I tried converting the dataset to excel, uploaded the same in Azure blob storage and tried fetching the required number of columns using the advanced excel option available by writing sql queries. By this method, I am able to get my output, but this does not work for a dataset which is huge.
I would be great if I can get a support to tackle this issue.