Source Klaviyo: added dbt converter

[parthiv11]

closes #17

[parthiv11]

@marcosmarxm just want to know, am I on right track?

[marcosmarxm]

Looks promising, @parthiv11. I'll review your contribution tomorrow and get back to you.

[parthiv11]

profiles' s -->person :- Transformation is performed with some nested JSON.
event ' s --> events' s :- Transformation is performed with some nested JSON.
metric' s --> metrics :- Transformation is performed with some nested JSON.

campaigns --> campaign with some unknow columns (probably joins with email template table)

Source Field	Mapped Field
campaign_type	type
created	attributes.created_at
email_template_id	❌
from_email	included[0].attributes.content.reply_to_email
from_name	included[0].attributes.label
id	id
name	attributes.name
send_time	attributes.send_time
sent_at	❌
status	attributes.status
status_label	attributes.status
status_id	❌
subject	included[0].attributes.content.subject
updated	attributes.updated_at
archived	attributes.archived
scheduled	attributes.scheduled_at

event--> events with some unknow columns

Source Field	Mapped Field
variation	included[0].relationships.flow-message-variation.data.id
campaign_id	included[0].relationships.flow.campaign.id
timestamp	attr.timestamp
flow_id	attributes.event_properties.$flow
flow_message_id	attributes.event_properties.$message
id	id
metric_id	relationships.metric.data.id
person_id	relationships.profile.data.id
type	type
uuid	attr.uuid
source_relation	-
klaviyo_source.remove_string_from_numeric('property_value')	??

ERD diagram

[parthiv11]

Looks promising, @parthiv11. I'll review your contribution tomorrow and get back to you.

@marcosmarxm

[marcosmarxm]

@parthiv11 let's continue without the columns and see if the model behave as expected.

[marcosmarxm]

@parthiv11 I pushed some changes. I was able to run using these models. For Klaviyo there is an small problem for staging which is different from other sources. It doesn't select a table but create an union. For that reason it won't be possible to run a simple dbt run... command to have all done, but it is possible to run:

dbt run --models +source_klaviyo --target postgres # build tables used by Fivetran this is the Airbyte models
dbt run --models +klaviyo_source --target postgres # run Fivetran models

Can you move from draft to ready to review? I think there only a couple of things to be done here.

[parthiv11]

@marcosmarxm just pushed a commit , please check it out,
and let me know, what next

[marcosmarxm]

Awesome work @parthiv11

[marcosmarxm]

I ran all tests manually for all 3 databases (postgres, snowflake and bigquery) It is working perfectly! Thanks @parthiv11

[parthiv11]

Thank you marcos, without your help I would not be able to do early

…

On Wed, Aug 28, 2024, 11:41 PM Marcos Marx ***@***.***> wrote: Merged #52 <#52> into main. — Reply to this email directly, view it on GitHub <#52 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASBGDTFGGPGR6SI32FQFXXLZTYHENAVCNFSM6AAAAABMOVL7L2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGA2TAOJWGU3TSMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>