Used the "OData Incremental Load" connector to successfully load a table from SAP Cloud with ~10k records. All fields present and contents look good. My incremental column is of data type TIMESTAMP_NTZ. When I re-run my job to pickup changed records it fails with "Issue occurred during query execution in TREX (&CODE&): &TEXT&|Issue occurred during query execution in TREX (6930 ): attribute value is not a number."
I know it's not a number, it's a timestamp - Trex is the SAP indexing engine so I assume it's throwing the error but I have no idea how to even begin troubleshooting in Matillion so any tips appreciated
Hi @PH398614, I believe I have heard of the OData Incremental Load component but it's not in any of the newer versions of Matillion. I see no mention of the component in the documentation. I am wondering if you are using perhaps an older version of Matillion that had the component which no longer exists in the newer versions? Could you please tell us which version of Matillion you are running and what cloud provider is hosting your Matillion instance? A year or two ago, this was important because the products and cloud providers the products were on didn't align perfectly from a feature perspective.
Just thinking through your post, makes me wonder if you are better off troubleshooting the error from the SAP side. The error you are getting is definitely something being generated from SAP. I am wondering if SAP logs requests that are coming in? Being able to look at that log will allow you to see what Matillion's request was to cause SAP to throw the error. This might lead you the resolution. Hopefully that helps! Thanks for posting!
Its a component I downloaded from https://exchange.matillion.com/s/exchange-job/a074G00001EmAsvQAF/odata-incremental-load.
Was trying it after I ran into same issue with the standard OData control which I was using with a "Query Result To Scalar" to get the max LastChangetDate from a table so I could just read new or changed records
Even when I hardcoded the date in the Date Source Filter of the standard OData control I can't get it to filter returned results. The Odata control appears to be unable to do anything except a full data load which isn't an option because of data volume and the fact that Odata is so slow.
I'm able to get deltas if I use an API Query but it's a lot to setup and maintain as I have about 20 sources I need to delta enable, Odata would have been much nicer
I had checked the Server Logs but there was nothing helpful other than the same error
One more thing to look at is the Matillion Server Logs. Often times you get some robust logging in the server logs that don't exist in the UI itself. You can download those by going to Admin button --> Download Server Log.
Hi Phil, sorry it took me so long to get back to you. I have been a bit under the weather. The only other option I can think of is to use Python. Unfortunately OData isn't a fun endpoint to work with and perhaps the reason why it's not used anymore in new implementations.
You might try reaching out to Matillion Support and see if they could offer any other approaches that I haven't thought of.