Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(analytics): Add Clickhouse based analytics #2988

Merged
merged 40 commits into from
Nov 29, 2023

Conversation

lsampras
Copy link
Member

@lsampras lsampras commented Nov 27, 2023

Type of Change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring
  • Dependency updates
  • Documentation
  • CI/CD

Description

  • Add kafka as a event destination for the generated events
  • Add events for changes to payment attempt / intent & refund tables
  • Add clickhouse support for powering the the /analytics API's

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

Motivation and Context

  • Postgres might not be the best solution out there to support OLAP queries
  • we've decided to provide CKH as an OOTB solution to run some of the OLAP queries to support the /analytics endpoint

How did you test it?

Events generated by Kafka

Events generated by kafka

Analytics OSS dashboard
Analytics dashboard powered by ckh

Checklist

  • I formatted the code cargo +nightly fmt --all
  • I addressed lints thrown by cargo clippy
  • I reviewed the submitted code
  • I added unit tests for my changes where possible
  • I added a CHANGELOG entry if applicable

@lsampras lsampras added A-framework Area: Framework C-feature Category: Feature request or enhancement labels Nov 27, 2023
@lsampras lsampras self-assigned this Nov 27, 2023
config/development.toml Show resolved Hide resolved
crates/analytics/Cargo.toml Show resolved Hide resolved
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SanchithHegde do we need to link the clickhouse contribution setup, to the contributing.md or any other root level documentation?

@lsampras lsampras marked this pull request as ready for review November 28, 2023 06:03
@lsampras lsampras requested review from a team as code owners November 28, 2023 06:03
@harsh-sharma-juspay
Copy link
Contributor

harsh-sharma-juspay commented Nov 28, 2023

curl --location --request GET 'http://localhost:8080/analytics/v1/api_event_logs?type=Payment&payment_id=pay_tPKkHpRnNealb3VTuLkC' \
--header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0' \
--header 'Accept: */*' \
--header 'Accept-Language: en-US,en;q=0.5' \
--header 'Accept-Encoding: gzip, deflate, br' \
--header 'Referer: https://app.hyperswitch.io/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoiMmU4ZTM4ZGMtNTQ1Ny00Nzk1LWI1ZWEtZmJmOWI5YzBhMTc1IiwibWVyY2hhbnRfaWQiOiJtZXJjaGFudF8xNzAxMTc2MjA5Iiwicm9sZV9pZCI6Im9yZ19hZG1pbiIsImV4cCI6MTcwMTM1MjA0Niwib3JnX2lkIjoib3JnX2hyQWVsRUhvbzlDb3lxcjJpM0JpIn0.IX7Lgj5UuRfMfoYx5M20qNkrvhp28VFqMmhoqL9TNeE' \
--header 'api-key: hyperswitch' \
--header 'Origin: https://app.hyperswitch.io' \
--header 'Connection: keep-alive' \
--header 'Sec-Fetch-Dest: empty' \
--header 'Sec-Fetch-Mode: cors' \
--header 'Sec-Fetch-Site: same-site' \
--header 'TE: trailers'
Screenshot 2023-11-28 at 7 21 22 PM

harsh-sharma-juspay and others added 5 commits November 28, 2023 19:23
Co-authored-by: akshay.s <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gnanasundari24 <[email protected]>
Co-authored-by: hyperswitch-bot[bot] <148525504+hyperswitch-bot[bot]@users.noreply.github.com>
ivor11
ivor11 previously approved these changes Nov 28, 2023
Copy link
Member

@vspecky vspecky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any routing changes, so approving to unblock required review from hyperswitch-routing

@likhinbopanna likhinbopanna added this pull request to the merge queue Nov 29, 2023
Merged via the queue into main with commit 9df4e01 Nov 29, 2023
20 of 22 checks passed
@likhinbopanna likhinbopanna deleted the analytics_oss_migration branch November 29, 2023 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-framework Area: Framework C-feature Category: Feature request or enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.