I have a Salesforce Incremental Load component working successfully in my Dev environment One of the steps has record count for updated and inserted rows.. I can see this component and its values in the task history for this job step. However, I want to extract this information and add to our audit table. Which component do I use to achieve this?
Here is a screenshot of what I see in task history of the value that I want to capture:
Your component "Adjust Logged Counts" is what type of component? Where do those 3 values come from?
Like is it a Python component which calls an API, and prints those 3 values?
Or is it a Salesforce Query component, and those 3 values come from the "Export" tab of the component?
I am not using Python, I am using the Salesforce Incremental Load component. I only see those values when I'm watching the component as it runs. I clicked on that step once the run was complete and saw these values, which I need.
Unfortunately, I don't see those values available in the "Export" tab of this component. I wish they were there, I would have used them. The only values in the "Export" tab are the ones I added which are our standard values and map to our local variables. Screenshot of the values I added. However, since this component using our variables, the row count is missing.
Am I able to add them in the "Export" tab even though they aren't our local variables?
Hi @kerri.gathright,
What cloud warehouse are you using (Snowflake, BigQuery, Redshift, etc.)? If you are using Snowflake there is a way to get the same 3 values using change tracking. Basically, you turn Change Tracking on for the table that the incremental changes are being applied to and then you can run a query like below which leverages change tracking and time travel:
select *
from yourtable
changes(information => default)
at(timestamp => '2021-05-28 00:00:00'::timestamp);
Here is the documentation for CHANGES which talks about this. Although this is more complicated than if you could just get the values out of the component, the positive is that the CHANGES method allows you to get true deltas at the destination which is likely to be more accurate. I hope this helps.
We are using Snowflake and this is a great suggestion. My company hasn't implemented this on any tables/schemas. Definitely worth a conversation. Thank you!
the screenshot didn't upload with my question post so I put it in the answers. Hope this doesn't cause any confusion.
