Hello the community, I have several files deposited in GCS, the problem is that some have an additional column (example some 5 columns and others 6 columns). Please how to remove last column from 6 column files

Matillion ETL
1

Hello the community,

 

I have several files deposited in GCS, the problem is that some have an additional column (example some 5 columns and others 6 columns). The 6 column files, I would like to delete the last column

 

i want to extract them from GCS and load them into a BigQuery database from cloud storage load component

 

And when it loads a 6 column file, it must delete the last column to have the same structure corresponding to the table schema (5 columns)

 

Please how to remove last column from 6 column files

 

2

Hi @tatus_parker​,

Neither Matillion ETL nor Matillion Data Loader has components for altering source files. In fact the normal approach is the opposite: load source data as-is, and then perform integration and format unification using transformations inside the target database (i.e. BigQuery in your case). It's better for audit and traceability that way.

So I would recommend loading the files as they are, and then removing the extra column in a Transformation Job where necessary.

Best regards,

Ian

3

could you not load the files with 6 columns then load the files with 5 columns (you could then ignore the 6th column in the second load)

 

4

Hi @ian.funnell​

I would like to load the source files as they are but the problem I have an error while loading the files because the files do not have the same structure.

How could I load these 2 types of files as they are in the same BigQuery table?

Best regards