Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7891] - Adapt stacktrace and exception formatting #4863

Conversation

d-a-v-e
Copy link
Contributor

@d-a-v-e d-a-v-e commented Dec 6, 2024

Context

For the last 2 Mondays (since 28 November), when the Dqt::SyncTeachersJob has run on production it has produced a very high volume of log entries (~9 million) which causes issues for our logit quota, and is clearly unnecessary.

Changes proposed in this pull request

Adapts the stack trace and exception formatting

Copy link

codecov bot commented Dec 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.21%. Comparing base (f30f535) to head (163eed7).
Report is 11 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4863   +/-   ##
=======================================
  Coverage   96.21%   96.21%           
=======================================
  Files         752      752           
  Lines       16204    16207    +3     
=======================================
+ Hits        15590    15593    +3     
  Misses        614      614           
Flag Coverage Δ
unittests 96.21% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@d-a-v-e d-a-v-e force-pushed the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch from 9013559 to b7d2dcb Compare December 9, 2024 14:19
@d-a-v-e d-a-v-e force-pushed the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch from 7323ac4 to 479e0fa Compare December 10, 2024 11:31
@tomtrentham tomtrentham marked this pull request as draft December 12, 2024 10:06
@tomtrentham tomtrentham changed the title [7891] - adapt stacktrace and exception formatting [7891] - Adapt stacktrace and exception formatting Jan 14, 2025
@d-a-v-e d-a-v-e force-pushed the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch from de34bba to 6498b21 Compare January 17, 2025 10:24
@d-a-v-e d-a-v-e added the deploy label Jan 17, 2025
@d-a-v-e d-a-v-e force-pushed the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch from eb29507 to a0a685a Compare January 17, 2025 12:19
@d-a-v-e d-a-v-e marked this pull request as ready for review January 17, 2025 12:19
@d-a-v-e d-a-v-e requested a review from tomtrentham January 22, 2025 09:40
@d-a-v-e d-a-v-e force-pushed the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch from 248db28 to 3727cc7 Compare January 22, 2025 12:23
Copy link
Contributor

@tomtrentham tomtrentham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of minor suggestions in the docs.

docs/logging.md Outdated
# Logging Documentation

## Rails Semantic Logger
We use `rails_semantic_logger` instead of the default Rails logger because it provides enhanced logging capabilities. It solves several problems such as:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We use `rails_semantic_logger` instead of the default Rails logger because it provides enhanced logging capabilities. It solves several problems such as:
We use [`rails_semantic_logger`](https://github.com/reidmorrison/rails_semantic_logger) instead of the default Rails logger because it provides enhanced logging capabilities. It solves several problems such as:

docs/logging.md Outdated
- [Production Logs](https://kibana-uk1.logit.io/s/e9b9162d-0b5e-4362-bed0-8e577f88d06e/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'filebeat-*',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:kubernetes.container.name,negate:!f,params:!(register-production,register-production-worker),type:phrases,value:'register-production,%20register-production-worker'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(kubernetes.container.name:register-production)),(match_phrase:(kubernetes.container.name:register-production-worker))))))),query:(language:kuery,query:'')))
- [QA Logs](https://kibana-uk1.logit.io/s/b2876053-0173-44fc-983d-99567292ba31/app/data-explorer/discover#?_a=(discover:(columns:!(kubernetes.container.name,app.message,app.payload.controller,url.path),isDirty:!t,sort:!()),metadata:(indexPattern:'filebeat-*',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-10m,to:now))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:app.name,negate:!t,params:(query:HeartbeatController),type:phrase),query:(match_phrase:(app.name:HeartbeatController))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:kubernetes.container.name,negate:!f,params:!(register-qa,register-qa-worker),type:phrases,value:'register-qa,%20register-qa-worker'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(kubernetes.container.name:register-qa)),(match_phrase:(kubernetes.container.name:register-qa-worker))))))),query:(language:kuery,query:'')))

**Note:** the two different GUID's used for Prod (production, productiondata) and Test(QA, staging, review apps)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Note:** the two different GUID's used for Prod (production, productiondata) and Test(QA, staging, review apps)
**Note:** the two different GUID's used for Prod (production, productiondata) and Test (QA, staging, review apps)

@d-a-v-e d-a-v-e merged commit b447ece into main Jan 23, 2025
1 of 2 checks passed
@d-a-v-e d-a-v-e deleted the 7891-investigate-and-fix-high-volume-of-logging-on-register-production-worker branch January 23, 2025 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants