I am ingesting NetSuite tables into our data warehouse. To support better Data quality I have configured the Netsuite component to run a custom query to retrieve a couple data points like: record count, max timestamp,... that I can check against after I have ingested the table. The issue is that some of the queries take multiple hours to run. Has anyone else run into an issue like this?
Example query:
SELECT NULL as "SOURCE_BUSINESS_CHECK_SUM", NULL as "SOURCE_FIRST_RECORD_CREATED", NULL as "SOURCE_LATEST_RECORD_CREATED", COUNT('InternalId') as "SOURCE_TABLE_ROW_COUNT", CONCAT('NETSUITE', '') as "SOURCE_SYSTEM_NAME", UPPER(CONCAT('JournalEntry_lineList', '')) as "SOURCE_TABLE_NAME", MAX("LastModifiedDate") as "SOURCE_WATERMARK_MAX" FROM "JournalEntry_lineList"
Note: I know the query is a little ugly, but it dose run and complete just takes a significant amount of time.
@nathan.lang1614377435484 can I ask you how you are connecting to NetSuite? Is this via NetSuite native connector inside Matillion?
I have an issue that I cannot get to the JournalEntry_lineList. There is no corresponding table named like that. I also have an issue getting to the Transaction_Lines.
Any help would be appriciated.
Borut
Hi Nathan,
thank you for your feedback. I have managed to connect to JournalEntry_lineList and now I am just struggling with the Transaction_lines :/.
Borut
Hey Borut,
Sorry for the delayed response to this post.
For Transaction Lines I believe we had to use "Transactions" which is a special Object that Matillion defines behind the scenes. I believe we also had issues with getting the level of detail needed for Transactions and eventually had to create a Saved Search and then ingest that to get to the detail needed. But you may be able to use "CustomTransaction_lineList". I am not a 100% on that I have not looked at that obj.
Hello Borut,
I am using the native connector to retrieve JournalEntry_lineList and Transaction records. You may need to add in a couple connection options the ones I am using are listed below:
