feat(config): update vector config

[maverox]

Type of Change

• Bugfix
• New feature
• Enhancement
• Refactoring
• Dependency updates
• Documentation
• CI/CD

Fixes #6364

Description

• adding a transform for sdk logs that fetches a value from an enrichment table sdk_map populated by file sdk_map.csv and updates the fields using this value.
• adding some fields to ignore/except in opensearch logs sink
  Important Notes
  Please ensure that the sdk_map.csv file is properly formatted and accessible at the specified path. The name of the file should also match the name of the table in config i.e sdk_map.

Additional Changes

• This PR modifies the API contract
• This PR modifies the database schema
• This PR modifies application configuration/environment variables

Motivation and Context

Prior Behaviour

• sdk logs have a merchant_id field but it was being populated with publishable_key as value at source
  observed merchant_id: "pk_dev*****".
  expected merchant_id: "merchant_*****".
• open search logs have certain fields like last_synced, sign_flag and clickhouse_database that needed to be ignored

Intentions

• Ensuring correctness of logs and reduce log size by storing only relevant fields.

How did you test it?

Running it locally to check logs added in console (sdk) logs & opensearch

sdk_logs

I created a dummy_transform event that inserted a publishable_key in merchant_id field and passed it to the actual
transform function and then captured the output on a
console sink

• test-code

• sdk_map.csv (with dummy value 19999)

• console output

opensearch logs

for this simply created a payment and compared the logs from kafka topics and opensearch entries

kafka ui

opensearch dashboard (no last_synced entry found)

Checklist

• I formatted the code cargo +nightly fmt --all
• I addressed lints thrown by cargo clippy
• I reviewed the submitted code
• I added unit tests for my changes where possible

[semanticdiff-com]

Review changes with  

[lsampras]

Do we need to add a placeholder csv?

[maverox]

Do we need to add a placeholder csv?

Yes for the enrichment to work I think we need csv but I am not sure what you mean by placeholder?