Matillion looking for use cases for syncing data back to DynamoDB

Matillion ETL
1

I am a member of the Matillion product team, and I am looking for use cases for a proposed DynamoDB Output connector. I am looking for details on:

 

  1. What platforms this would be useful for (Snowflake/DeltaLake/RedShift/Synapse)?
  2. What is your business use case for this functionality?
  3. What features / capabilities would be vital for this connector to be considered successful?

Thank you for any feedback you may be able to give.

2

Hello, we have one use case.

 

1) Snowflake/AWS

 

2) We have API's that use DynamoDB as their main operational database. We have a daily/monthly synchronization of some metadata stored in Snowflake towards DynamoDB. This is at the moment done as csv -> s3, and lambda s3 -> dynamoDB. This works fine, but would be awesome to be able to directly throw data into DynamoDB as per defined.

 

3) I would guess a basic functionality to have a table -> column mapping, as 'columnar' as possible.

And with more advanced, maybe generate some JSON with some various depth dependent on requirements.

 

I can give you more examples if required :)

3

Hello Michael

 

Thank you very much for your response.

Can I ask that you further elaborate on, "generate some JSON with some various depth dependent on requirements"?

 

Regards

Ben

Matillion Product Team

4

Hiya,

 

I guess creating a nested json by the fly was the one I was after there :)

 

Example,

 

Let's say a table,

ID|Product_ID|Product_Name|Effective_Start|Effective_End|Effective_Price

1234|321321|Product Name Example #1|2020-01-01T00:00:00.000Z|2020-02-01T00:00:00.000Z|100.00

1234|321321|Product Name Example #1|2020-02-02T00:00:00.000Z|2999-01-01T00:00:00.000Z|99.00

 

The json structure I would want from this is,

 

 

{

"ProductList": [

{

"ID": {

"N": "1234"

},

"Product_ID": {

"N": "321321"

},

"Product_Name": {

"S": "Product Name Example #1"

},

"Effective": [

{

"Effective_Start": {

"S": "2020-01-01T00:00:00.000Z"

},

"Effective_End": {

"S": "2020-02-01T00:00:00.000Z"

},

"Effective_Price": {

"N": "100.00"

}

},

{

"Effective_Start": {

"S": "2020-02-02T00:00:00.000Z"

},

"Effective_End": {

"S": "2999-01-01T00:00:00.000Z"

},

"Effective_Price": {

"N": "99.00"

}

}

]

}

]

}