We have a CDC process setup to pull from Oracle to SF. Every so often this table does a purge of older data. This translates to a CDC file of about 130MB in the S3 cdc bucket with the record type 'D' meaning Delete in the target.
But what seems to be happening is that before this file is even uploaded to S3/added to SQS/picked up by Matillion, a newer change file (of a few KB) gets pushed through.
As a result, when the 130MB change file finally comes through Matillion complains that the Incoming file xxxx.csv is before last load file! 2022-06-11 05:55:34.943 >= 2022-06-11 05:50:34.295000.
Has anyone run into this issue before?
TIA!!