Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add repository metrics #177

Open
wants to merge 20 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
6cebdbb
First version of integrated repository metrics
steffen Feb 27, 2019
d5320c7
Add passing around nwo in query string in navigation links
steffen Feb 27, 2019
69b3577
Add loading repository specific data
steffen Feb 27, 2019
2480fd4
Add back to instance statistics link to repo view
steffen Feb 28, 2019
4495986
Add overview table to overview page
steffen Feb 28, 2019
5b33604
Add first version of creating a report for monitored repositories
steffen Mar 1, 2019
b68d0a3
Add report for GitHub api request types by count
steffen Mar 1, 2019
0d2b923
Add abstract ReportRepository class
steffen Mar 1, 2019
93c8f0f
Add monitored repositories
steffen Mar 1, 2019
06384ab
Add showing repository name in page header
steffen Mar 1, 2019
13ee2ec
Add writing repository name directly into the header to avoid flickering
steffen Mar 1, 2019
d9495fd
Add git-sizer metrics
steffen Apr 5, 2019
44b62be
Make "passing" repository to scripts work
steffen Apr 5, 2019
221fafc
Exclude internal api requests from api request types data
steffen Apr 5, 2019
a75e34a
Add repository metrics support for non-remote runs
steffen Apr 17, 2019
654b75b
Add repository Git traffic metrics
steffen Apr 17, 2019
7008ec9
Consolidate git-sizer report and include all quantity values
steffen Apr 17, 2019
189d46c
Tweaked charts to focus on actionable metrics
steffen Apr 18, 2019
c1b1a8a
Make updater scripts executable
steffen May 9, 2019
eff5fca
Fix issue with relative dataURLs by only supporting absolute dataURLs
steffen Aug 18, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add report for GitHub api request types by count
  • Loading branch information
steffen committed Mar 1, 2019
commit b68d0a3d1367583324d7e4afb232739badde7187
21 changes: 21 additions & 0 deletions updater/reports/repository/ReportGitHubApiRequestTypesByCount.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from ..Report import *

# Lists the number of GitHub api request types
# Ordered by count in descending order
class ReportGitHubApiRequestTypesByCount(Report):
def name(self):
return "github-api-request-types-by-count"

def metaName(self):
return self.repository + "/" + self.name()

def fileName(self):
return os.path.join(self.dataDirectory, "repository", self.repositoryOwner, self.repositoryName, self.name() + ".tsv")

# The data is overwritten every day, so skip reading the old data
def readData(self):
pass

def updateData(self):
self.header, self.data = self.parseData(
self.executeScript(self.scriptPath("repository/github-api-request-types-by-count.sh")))
3 changes: 1 addition & 2 deletions updater/reports/repository/ReportOverview.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
from ..Report import *

# Lists the number of contributors for each organization
# Ordered by the number of contributors in descending order
# Lists the name and the size of the repository
class ReportOverview(Report):
def name(self):
return "overview"
19 changes: 19 additions & 0 deletions updater/scripts/repository/github-api-request-types-by-count.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/bash

#
# Count number of API calls per route
#

echo -e "type\tcount"

zcat -f /var/log/github/unicorn.log.1* |
grep -F 'request_category=api' |
grep -Fv 'remote_address=127.0.0.1' |
grep "$1" |
grep -oP 'route=\K\S+' |
grep -Fvx 'nil' |
sort |
uniq -ic |
sort -rn |
head -20 |
awk '{ printf("%s\t%s\n", $2, $1) }'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks pretty similar to https://github.com/Autodesk/hubble/blob/master/updater/scripts/api-requests-by-user.sh ... should we add a repo option to the original script to avoid code duplication?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! There are other reports I wanted to add for repo metrics that are pretty much the same as well. I'll explore adding a repo option to the original scripts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@larsxschneider I added a repo option for git-download.sh. Let me know if you think that's a good solution and if it's ok to use for other reports/scripts as well.

2 changes: 2 additions & 0 deletions updater/update-stats.py
Original file line number Diff line number Diff line change
@@ -35,6 +35,7 @@
from reports.ReportTokenlessAuth import *
from reports.ReportUsers import *
from reports.repository.ReportOverview import *
from reports.repository.ReportGitHubApiRequestTypesByCount import *

def writeMeta(dataDirectory):
outputFilePath = os.path.join(dataDirectory, "meta.tsv")
@@ -106,6 +107,7 @@ def main():
# Repository reports
for repository in configuration["monitoredRepositories"]:
ReportOverview(configuration, dataDirectory, metaStats, repository).update()
ReportGitHubApiRequestTypesByCount(configuration, dataDirectory, metaStats, repository).update()

# Write meta infos
writeMeta(dataDirectory)