Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: basic CI/CD were added #16

Merged
merged 28 commits into from
Dec 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f28f706
feat: basic CI/CD were added
scientiststwin Dec 14, 2023
99137eb
feat: docker-compose.test.yml file was added
scientiststwin Dec 14, 2023
8641727
Merge branch 'main' into feature/ci
scientiststwin Dec 15, 2023
c174a10
feat: two unneeded files were removed
scientiststwin Dec 15, 2023
4abba63
feat: unneeded hear was removed
scientiststwin Dec 15, 2023
e08e266
feat: align code with Linter
scientiststwin Dec 15, 2023
c15d7ca
feat: more lint formate
scientiststwin Dec 15, 2023
24f78df
feat: more lint formate
scientiststwin Dec 15, 2023
9587695
feat: ignore __init__.py files from liniting
scientiststwin Dec 15, 2023
74f931c
feat: resolved
scientiststwin Dec 15, 2023
66e6cf9
feat: docker-compose.test.yml file updated align with github ci/cd
scientiststwin Dec 15, 2023
fb5b7e4
feat: new docker-compose file was written
scientiststwin Dec 15, 2023
5932d95
fix: manage code in a way that can clean the bash
scientiststwin Dec 15, 2023
7c0f069
feat: a basic unit test were added for saving labels into db
scientiststwin Dec 18, 2023
1efb76b
feat: a dockerfile & docker-compose & etc were added for managin test…
scientiststwin Dec 18, 2023
00fd27c
fix: lint fixes
scientiststwin Dec 18, 2023
bc5befa
feat: aligned with lint
scientiststwin Dec 18, 2023
c9d2254
feat: docker-entrypoint.sh updated
scientiststwin Dec 18, 2023
d3444ff
fix: update docker-entrypoint.sh to save all changes
scientiststwin Dec 18, 2023
b1f7e6b
feat: add 'coverage' file to the .gitignore file
scientiststwin Dec 18, 2023
b721896
feat: test staged was added to Dockerfile and docker-compose
scientiststwin Dec 18, 2023
5a2708c
fix: typo was fixed
scientiststwin Dec 18, 2023
174289d
fix: remove production.yml file as we don't need that for now
scientiststwin Dec 18, 2023
b520863
Merge branch 'main' into feature/ci
scientiststwin Dec 19, 2023
fdd2258
feat: add prod stage to the Dockerfile
scientiststwin Dec 19, 2023
464137a
feat: aligned codes with lint
scientiststwin Dec 19, 2023
6eacf8e
feat: aligned with lint
scientiststwin Dec 19, 2023
3672891
feat: short the lines
scientiststwin Dec 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/start.staging.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: Staging CI/CD Pipeline

on: pull_request

jobs:
ci:
uses: TogetherCrew/operations/.github/workflows/ci.yml@main
secrets:
CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ coverage.xml
.hypothesis/
.pytest_cache/
cover/
coverage

# Translations
*.mo
Expand Down
12 changes: 12 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# It's recommended that we use `bullseye` for Python (alpine isn't suitable as it conflcts with numpy)
FROM python:3.10-bullseye AS base
WORKDIR /project
COPY . .
RUN pip3 install -r requirements.txt

FROM base AS test
RUN chmod +x docker-entrypoint.sh
CMD ["./docker-entrypoint.sh"]

FROM base AS prod
CMD ["echo", "aiflow dags should be running on airlfow container"]
5 changes: 2 additions & 3 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,16 @@


## Running the app

You can quickly launch the application using `Docker Compose`:

```bash
docker-compose --profile flower up
```

## Lint the code

## Lint the code
The code can be linted by using the below command

```bash
python -m black .
```
```
40 changes: 0 additions & 40 deletions dags/businees-requremensts.txt

This file was deleted.

45 changes: 23 additions & 22 deletions dags/github.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,42 +17,41 @@
# under the License.
"""Example DAG demonstrating the usage of dynamic task mapping."""
from __future__ import annotations

from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task

from github_api_helpers import (
get_all_repo_issues_and_prs_comments,
get_all_reviews_of_pull_request,
get_all_repo_review_comments,
get_all_repo_contributors,
get_all_pull_request_files,
get_all_pull_requests,
get_all_org_members,
get_all_repo_labels,
get_all_org_repos,
get_all_org_repos,
fetch_org_details,
fetch_commit_files,
fetch_org_details,
get_all_commits,
get_all_issues,
get_all_org_members,
get_all_org_repos,
get_all_pull_request_files,
get_all_pull_requests,
get_all_repo_contributors,
get_all_repo_issues_and_prs_comments,
get_all_repo_labels,
get_all_repo_review_comments,
get_all_reviews_of_pull_request,
)
from neo4j_storage import (
save_commit_files_changes_to_neo4j,
save_repo_contributors_to_neo4j,
save_pr_files_changes_to_neo4j,
save_review_comment_to_neo4j,
get_orgs_profile_from_neo4j,
save_pull_request_to_neo4j,
save_org_member_to_neo4j,
save_comment_to_neo4j,
save_commit_files_changes_to_neo4j,
save_commit_to_neo4j,
save_review_to_neo4j,
save_issue_to_neo4j,
save_label_to_neo4j,
save_org_member_to_neo4j,
save_orgs_to_neo4j,
save_pr_files_changes_to_neo4j,
save_pull_request_to_neo4j,
save_repo_contributors_to_neo4j,
save_repo_to_neo4j,
save_review_comment_to_neo4j,
save_review_to_neo4j,
)

with DAG(
Expand All @@ -67,18 +66,20 @@ def get_all_organization():
orgs = get_orgs_profile_from_neo4j()
return orgs

#! for testing
# !for testing
# toghether_crew_org = {
# "id": 1,
# "name": "TogetherCrew",
# "description": "TogetherCrew is a community of developers, designers, and creators who are passionate about building and learning together.",
# "description": """TogetherCrew is a community of developers, designers, and creators
# who are passionate about building and learning together.""",
# "url": "",
# "key": ""
# }
# rndao_org = {
# "id": 2,
# "name": "RnDAO",
# "description": "RnDAO is a community of developers, designers, and creators who are passionate about building and learning together.",
# "description": """RnDAO is a community of developers, designers, and creators
# who are passionate about building and learning together.""",
# "url": "",
# "key": ""
# }
Expand Down
23 changes: 12 additions & 11 deletions dags/github_api_helpers/__init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
from .repos import get_all_org_repos, get_all_repo_contributors
from .commits import get_all_commits, fetch_commit_details, fetch_commit_files
from .issues import get_all_issues, get_all_comments_of_issue
# flake8: noqa
from .comments import get_all_repo_issues_and_prs_comments, get_all_repo_review_comments
from .commits import fetch_commit_details, fetch_commit_files, get_all_commits
from .issues import get_all_comments_of_issue, get_all_issues
from .labels import get_all_repo_labels
from .orgs import fetch_org_details, get_all_org_members
from .pull_requests import (
get_all_pull_requests,
get_all_commits_of_pull_request,
get_all_comments_of_pull_request,
get_all_review_comments_of_pull_request,
get_all_reactions_of_review_comment,
get_all_commits_of_pull_request,
get_all_pull_request_files,
get_all_pull_requests,
get_all_reactions_of_comment,
get_all_reactions_of_review_comment,
get_all_review_comments_of_pull_request,
get_all_reviews_of_pull_request,
get_all_pull_request_files,
)
from .orgs import fetch_org_details, get_all_org_members
from .labels import get_all_repo_labels
from .comments import get_all_repo_review_comments, get_all_repo_issues_and_prs_comments
from .repos import get_all_org_repos, get_all_repo_contributors
13 changes: 11 additions & 2 deletions dags/github_api_helpers/comments.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging

from .smart_proxy import get


Expand Down Expand Up @@ -45,7 +46,11 @@ def fetch_repo_review_comments_page(
)
)

logging.info(f"Found {len(updated_response_data)} comments for {owner}/{repo} on page {page}. comments: {updated_response_data}")
msg = f"Found {len(updated_response_data)}"
msg += f" review comments for {owner}/{repo} on page {page}."
msg += f" comments: {updated_response_data}"
logging.info(msg)

return updated_response_data


Expand Down Expand Up @@ -117,7 +122,11 @@ def fetch_repo_issues_and_prs_comments_page(
map(lambda x: {**x, **extract_type_from_comment_response(x)}, response_data)
)

logging.info(f"Found {len(updated_response_data)} comments for {owner}/{repo} on page {page}. comments: {updated_response_data}")
msg = f"Found {len(updated_response_data)}"
msg += f" comments for {owner}/{repo} on page {page}."
msg += f" comments: {updated_response_data}"
logging.info(msg)

return updated_response_data


Expand Down
13 changes: 10 additions & 3 deletions dags/github_api_helpers/commits.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging

from .smart_proxy import get


Expand All @@ -18,7 +19,9 @@ def fetch_commits(owner: str, repo: str, page: int, per_page: int = 100):
response = get(endpoint, params=params)
response_data = response.json()

logging.info(f"Found {len(response_data)} commits for {owner}/{repo} on page {page}. Commits: {response_data}")
logging.info(
f"Found {len(response_data)} commits for {owner}/{repo} on page {page}. Commits: {response_data}"
)
return response_data


Expand Down Expand Up @@ -61,7 +64,9 @@ def fetch_commit_details(owner: str, repo: str, commit_sha: str):
response = get(endpoint)
response_data = response.json()

logging.info(f"Found details for commit {commit_sha} of {owner}/{repo}: {response_data}")
logging.info(
f"Found details for commit {commit_sha} of {owner}/{repo}: {response_data}"
)
return response_data


Expand All @@ -77,7 +82,9 @@ def fetch_commit_files(owner: str, repo: str, sha: str):
logging.info(f"Fetching files changed in commit {sha} of {owner}/{repo}...")
commit_details = fetch_commit_details(owner, repo, sha)
if "files" in commit_details:
logging.info(f"Found {len(commit_details['files'])} files changed in commit {sha} of {owner}/{repo}.")
logging.info(
f"Found {len(commit_details['files'])} files changed in commit {sha} of {owner}/{repo}."
)
return commit_details["files"]
else:
logging.info(f"No files changed in commit {sha} of {owner}/{repo}.")
Expand Down
17 changes: 12 additions & 5 deletions dags/github_api_helpers/issues.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging

from .smart_proxy import get


Expand Down Expand Up @@ -27,7 +28,9 @@ def fetch_issues(owner: str, repo: str, page: int, per_page: int = 100):
issues = [issue for issue in response_data if "pull_request" not in issue]
is_more_issues = len(response_data) == per_page

logging.info(f"Found {len(issues)} issues for {owner}/{repo} on page {page}. Issues: {issues}")
logging.info(
f"Found {len(issues)} issues for {owner}/{repo} on page {page}. Issues: {issues}"
)
return issues, is_more_issues


Expand Down Expand Up @@ -78,7 +81,9 @@ def fetch_issue_comments(
response = get(endpoint, params=params)
response_data = response.json()

logging.info(f"Found {len(response_data)} comments for issue {issue_number} on page {page}. Comments: {response_data}")
logging.info(
f"Found {len(response_data)} comments for issue {issue_number} on page {page}. Comments: {response_data}"
)
return response_data


Expand All @@ -96,11 +101,13 @@ def get_all_comments_of_issue(owner: str, repo: str, issue_number: int):
current_page = 1
while True:
logging.info(f"Fetching page {current_page} of comments...")
comments = fetch_pull_request_comments(owner, repo, issue_number, current_page)
comments = fetch_issue_comments(owner, repo, issue_number, current_page)
if not comments: # Break the loop if no more comments are found
break
all_comments.extend(comments)
current_page += 1

logging.info(f"Found a total of {len(all_comments)} comments for issue {issue_number}.")

logging.info(
f"Found a total of {len(all_comments)} comments for issue {issue_number}."
)
return all_comments
8 changes: 6 additions & 2 deletions dags/github_api_helpers/labels.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from .smart_proxy import get
import logging

from .smart_proxy import get


def fetch_repo_labels_page(owner: str, repo: str, page: int, per_page: int = 100):
"""
Fetches the labels for a specific repository in GitHub.
Expand All @@ -17,7 +19,9 @@ def fetch_repo_labels_page(owner: str, repo: str, page: int, per_page: int = 100
response = get(endpoint, params=params)
response_data = response.json()

logging.info(f"Found {len(response_data)} labels for {owner}/{repo} on page {page}. Labels: {response_data}")
logging.info(
f"Found {len(response_data)} labels for {owner}/{repo} on page {page}. Labels: {response_data}"
)
return response_data


Expand Down
5 changes: 4 additions & 1 deletion dags/github_api_helpers/orgs.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging

from .smart_proxy import get


Expand Down Expand Up @@ -34,7 +35,9 @@ def fetch_org_members_page(org: str, page: int, per_page: int = 100):
response = get(endpoint, params=params)
response_data = response.json()

logging.info(f"Found {len(response_data)} members for organization {org} on page {page}. Members: {response_data}")
logging.info(
f"Found {len(response_data)} members for organization {org} on page {page}. Members: {response_data}"
)
return response_data


Expand Down
Loading