Couldn't get an updated record by using JDBC Incremental Load component

Matillion ETL
1

Hello, all.

​

I'm trying to load an updated record from MySQL to BQ by using JDBC Incremental Load component.

For example, there is a user table and structure is below.

​

id

user_code

name

nationality

phone_number

created_date

updated_date

​

Auto Increment = id, primary key = id, key = user_code

​

First, I run JBBC Incremental Load and it was success.

Then, I updated one record's column(ex. name: Jones to Bryant) and run component again, but component's log showed that there is no changed and skipping load of data.

Absolutely the record were updated in MySQL and I checked it surely, but when I run component many times, the result doesn't change.

​

Official document shows that Incremental Load tools can load records that are new or changed since the previous load but in my case, no longer changed.

​

If there is something wrong procedure, could you point out to it please?

 

Thanks

​

​

2

Hi @DK852652​,

I am going to make some assumptions and throw a theory out there that might help you. I have never used this component but I believe I understand how it works under the hood. Go back through the setup wizard and for the name column set it to incremental. Next run back through your test scenario where you up the name column and see if it ends up in the target, which I suspect it will. What's going on here is that Matillion is looking for something that you have identified that will/could change. Since you didn't identify that the column name will change then it assumes nothing changed. Basically, Matillion is generating a hash using all fields you marked as incremental on the source and then compares that hash to the hash on the target. If the hash across those fields is different from source to target then Matillion knows the data changed and the target needs to be updated.

In your case you specified the id (or user_code?) as incremental and since you changed name it thinks no update was made. I hope this helps. Post back and let us know if this helped or if I missed the mark and I will give it another shot. Thank you!

3

Dear @Bryan​

​

Hi.

Thank you for your kind advise.

​

I tried to set incremental and column key to name column but it doesn't change for original column. Instead new record was inserted to BQ.

Example is below.

​

  1. I run JDBC Incremental Load and then new table was created on BQ and all of data were inserted to table.
  2. I updated one record's column(ex. name: Jones to Bryant) and run component again.
  3. Log shows that there is one changes and insert it.
  4. On BQ, there is new record that name column is Bryant. In addition, the record that name column is Jones still exists.

​

My expectation is that new record doesn't insert, but only original record does change.

Is there any way to do like as I expect?

Thank you.

4

Dear @Bryan​

​

Yes, I feel like it is getting closer.

Thank you for your advise and kind reply even though you're Snowflake customer.

​

I asked this problem to Matillion Official Support and they demanded me exported project, so I sent it to them but after that there is no reply.

That's why I posted this on Matillion Community.

@MatillionProductTeam​

Could you advise me how to update the record to BQ.

Thank you.

​

5

Hello,

 

I am new to matilion and running into the same problem. For some of my incremental load - some records didn't copy over for some reason after numerous runs. Does anyone know there is a solution for using JDBC component?

6

It sounds like you are getting closer but still not quite as expected. Based on the documentation, it should work unless I am misunderstanding the docs. Since we don't use this component and we are a Snowflake customer, I am sort of at the end of my expertise. Perhaps someone else could help? @MatillionProductTeam​ @ian.funnell​