Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: support run without aws #597

Merged
merged 2 commits into from
Dec 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/aws-preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ jobs:
WebUrl="https://petercat.ai" \
StaticUrl="https://static.petercat.ai" \
PetercatEnv="preview" \
AWSGithubSecretName=${{ secrets.AWS_GITHUB_SECRET_NAME }} \
AWSStaticSecretName=${{ secrets.AWS_STATIC_SECRET_NAME }} \
AWSLLMTokenSecretName=${{ vars.AWS_LLM_TOKEN_SECRET_NAME }} \
AWSLLMTokenPublicName=${{ vars.AWS_LLM_TOKEN_PUBLIC_NAME }} \
AWSStaticKeyPairId=${{ secrets.AWS_STATIC_KEYPAIR_ID }} \
GithubSecretName=${{ secrets.X_GITHUB_SECRET_NAME }} \
StaticSecretName=${{ secrets.STATIC_SECRET_NAME }} \
LLMTokenSecretName=${{ vars.LLM_TOKEN_SECRET_NAME }} \
LLMTokenPublicName=${{ vars.LLM_TOKEN_PUBLIC_NAME }} \
StaticKeyPairId=${{ secrets.STATIC_KEYPAIR_ID }} \
S3TempBucketName=${{ vars.S3_TEMP_BUCKET_NAME }} \
GitHubAppID=${{ secrets.X_GITHUB_APP_ID }} \
GithubAppsClientId=${{ secrets.X_GITHUB_APPS_CLIENT_ID }} \
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/aws-prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ jobs:
WebUrl="https://petercat.ai" \
StaticUrl="https://static.petercat.ai" \
PetercatEnv="production" \
AWSGithubSecretName=${{ secrets.AWS_GITHUB_SECRET_NAME }} \
AWSStaticSecretName=${{ secrets.AWS_STATIC_SECRET_NAME }} \
AWSLLMTokenSecretName=${{ vars.AWS_LLM_TOKEN_SECRET_NAME }} \
AWSLLMTokenPublicName=${{ vars.AWS_LLM_TOKEN_PUBLIC_NAME }} \
AWSStaticKeyPairId=${{ secrets.AWS_STATIC_KEYPAIR_ID }} \
GithubSecretName=${{ secrets.X_GITHUB_SECRET_NAME }} \
StaticSecretName=${{ secrets.STATIC_SECRET_NAME }} \
LLMTokenSecretName=${{ vars.LLM_TOKEN_SECRET_NAME }} \
LLMTokenPublicName=${{ vars.LLM_TOKEN_PUBLIC_NAME }} \
StaticKeyPairId=${{ secrets.STATIC_KEYPAIR_ID }} \
S3TempBucketName=${{ vars.S3_TEMP_BUCKET_NAME }} \
GitHubAppID=${{ secrets.X_GITHUB_APP_ID }} \
GithubAppsClientId=${{ secrets.X_GITHUB_APPS_CLIENT_ID }} \
Expand Down
10 changes: 5 additions & 5 deletions README.en-US.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,11 +104,11 @@ The project requires environment variables to be set:
| `WEB_URL` | Required | Domain of the frontend web service | `https://petercat.ai` |
| `STATIC_URL` | Required | Static resource domain | `https://static.petercat.ai` |
| **AWS Related Environment Variables** |
| `AWS_GITHUB_SECRET_NAME` | Required | AWS secret file name | `prod/githubapp/petercat/pem` |
| `AWS_STATIC_SECRET_NAME` | Optional | The name of the AWS-managed CloudFront private key. If configured, CloudFront signed URLs will be used to protect your resources. For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html). | `prod/petercat/static` |
| `AWS_LLM_TOKEN_SECRET_NAME` | Optional | The name of the LLM signing private key managed by AWS. If configured, Petercat will use the RSA algorithm to manage the user's LLM Token. | `prod/petercat/llm` |
| `AWS_LLM_TOKEN_PUBLIC_NAME` | Optional | The name of the LLM signing public key managed by AWS. If configured, Petercat will use the RSA algorithm to manage the user's LLM Token. | `prod/petercat/llm/pub` |
| `AWS_STATIC_KEYPAIR_ID` | Optional | The Key Pair ID for AWS CloudFront. If configured, CloudFront signed URLs will be used to protect your resources. For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html). | `APKxxxxxxxx` |
| `X_GITHUB_SECRET_NAME` | Required | AWS secret file name | `prod/githubapp/petercat/pem` |
| `STATIC_SECRET_NAME` | Optional | The name of the AWS-managed CloudFront private key. If configured, CloudFront signed URLs will be used to protect your resources. For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html). | `prod/petercat/static` |
| `LLM_TOKEN_SECRET_NAME` | Optional | The name of the LLM signing private key managed by AWS. If configured, Petercat will use the RSA algorithm to manage the user's LLM Token. | `prod/petercat/llm` |
| `LLM_TOKEN_PUBLIC_NAME` | Optional | The name of the LLM signing public key managed by AWS. If configured, Petercat will use the RSA algorithm to manage the user's LLM Token. | `prod/petercat/llm/pub` |
| `STATIC_KEYPAIR_ID` | Optional | The Key Pair ID for AWS CloudFront. If configured, CloudFront signed URLs will be used to protect your resources. For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html). | `APKxxxxxxxx` |
| `S3_TEMP_BUCKET_NAME` | Required | AWS S3 bucket for temporary image files | `xxx-temp` |
| `SQS_QUEUE_URL` | Required | AWS SQS queue URL | `https://sqs.ap-northeast-1.amazonaws.com/xxx/petercat-task-queue` |
| **Supabase Related Environment Variables** |
Expand Down
10 changes: 5 additions & 5 deletions README.ja-JP.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,11 @@
| `WEB_URL` | 必須 | フロントエンドウェブサービスのドメイン | `https://petercat.ai` |
| `STATIC_URL` | 必須 | 静的リソースドメイン | `https://static.petercat.ai` |
| **AWS関連環境変数** |
| `AWS_GITHUB_SECRET_NAME` | 必須 | AWSシークレットファイル名 | `prod/githubapp/petercat/pem` |
| `AWS_STATIC_SECRET_NAME` | オプション | AWSが管理するCloudFrontのプライベートキーの名前。設定されている場合、CloudFrontの署名付きURLが使用され、リソースが保護されます。詳細については、[AWSドキュメント](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)を参照してください。 | `prod/petercat/static` |
| `AWS_LLM_TOKEN_SECRET_NAME` | オプション | AWSが管理するLLM署名プライベートキーの名前。設定されている場合、PetercatはRSAアルゴリズムを使用してユーザーのLLMトークンを管理します。 | `prod/petercat/llm` |
| `AWS_LLM_TOKEN_PUBLIC_NAME` | オプション | AWSが管理するLLM署名公開キーの名前。設定されている場合、PetercatはRSAアルゴリズムを使用してユーザーのLLMトークンを管理します。 | `prod/petercat/llm/pub` |
| `AWS_STATIC_KEYPAIR_ID` | オプション | AWS CloudFrontのキーID。設定されている場合、CloudFrontの署名付きURLが使用され、リソースが保護されます。詳細については、[AWSドキュメント](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)を参照してください。 | `APKxxxxxxxx` |
| `X_GITHUB_SECRET_NAME` | 必須 | AWSシークレットファイル名 | `prod/githubapp/petercat/pem` |
| `STATIC_SECRET_NAME` | オプション | AWSが管理するCloudFrontのプライベートキーの名前。設定されている場合、CloudFrontの署名付きURLが使用され、リソースが保護されます。詳細については、[AWSドキュメント](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)を参照してください。 | `prod/petercat/static` |
| `LLM_TOKEN_SECRET_NAME` | オプション | AWSが管理するLLM署名プライベートキーの名前。設定されている場合、PetercatはRSAアルゴリズムを使用してユーザーのLLMトークンを管理します。 | `prod/petercat/llm` |
| `LLM_TOKEN_PUBLIC_NAME` | オプション | AWSが管理するLLM署名公開キーの名前。設定されている場合、PetercatはRSAアルゴリズムを使用してユーザーのLLMトークンを管理します。 | `prod/petercat/llm/pub` |
| `STATIC_KEYPAIR_ID` | オプション | AWS CloudFrontのキーID。設定されている場合、CloudFrontの署名付きURLが使用され、リソースが保護されます。詳細については、[AWSドキュメント](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)を参照してください。 | `APKxxxxxxxx` |
| `S3_TEMP_BUCKET_NAME` | 必須 | 一時的な画像ファイル用のAWS S3バケット | `xxx-temp` |
| `SQS_QUEUE_URL` | 必須 | AWS SQSキューURL | `https://sqs.ap-northeast-1.amazonaws.com/xxx/petercat-task-queue` |
| **Supabase関連環境変数** |
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,11 @@
| `WEB_URL` | 必选 | 前端 Web 服务的域名 | `https://petercat.ai`
| `STATIC_URL` | 必选 | 静态资源域名 | `https://static.petercat.ai`
| **AWS 相关环境变量** |
| `AWS_GITHUB_SECRET_NAME` | 必选 | AWS 托管的 Github 私钥文件名 | `prod/githubapp/petercat/pem`
| `AWS_STATIC_SECRET_NAME` | 可选 | AWS 托管的 CloudFront 签名私钥名称。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `prod/petercat/static` |
| `AWS_LLM_TOKEN_SECRET_NAME` | 可选 | AWS 托管的 llm 签名私钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm` |
| `AWS_LLM_TOKEN_PUBLIC_NAME` | 可选 | AWS 托管的 llm 签名公钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm/pub` |
| `AWS_STATIC_KEYPAIR_ID` | 可选 | AWS CloudFront 的 Key Pair ID。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `APKxxxxxxxx` |
| `X_GITHUB_SECRET_NAME` | 必选 | AWS 托管的 Github 私钥文件名 | `prod/githubapp/petercat/pem`
| `STATIC_SECRET_NAME` | 可选 | AWS 托管的 CloudFront 签名私钥名称。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `prod/petercat/static` |
| `LLM_TOKEN_SECRET_NAME` | 可选 | AWS 托管的 llm 签名私钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm` |
| `LLM_TOKEN_PUBLIC_NAME` | 可选 | AWS 托管的 llm 签名公钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm/pub` |
| `STATIC_KEYPAIR_ID` | 可选 | AWS CloudFront 的 Key Pair ID。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `APKxxxxxxxx` |
| `S3_TEMP_BUCKET_NAME` | 可选 | 用于托管 AWS 临时图片文件 S3 的 bucket | `xxx-temp`
| `SQS_QUEUE_URL`| 必选 | AWS SQS 消息队列 URL | `https://sqs.ap-northeast-1.amazonaws.com/xxx/petercat-task-queue`
| **SUPABASE 相关 env** |
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/self_hosted_aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ sam deploy \
--config-file .aws/petercat-ap-southeast.toml \
--parameter-overrides APIUrl=$API_URL \
WebUrl=$WEB_URL \
AWSSecretName=$AWS_GITHUB_SECRET_NAME \
AWSSecretName=$X_GITHUB_SECRET_NAME \
S3TempBucketName=$S3_TEMP_BUCKET_NAME \
GitHubAppID=$X_GITHUB_APP_ID \
GithubAppsClientId=$X_GITHUB_APPS_CLIENT_ID \
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/self_hosted_aws_cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ sam deploy \
--config-file .aws/petercat-ap-southeast.toml \
--parameter-overrides APIUrl=$API_URL \
WebUrl=$WEB_URL \
AWSSecretName=$AWS_GITHUB_SECRET_NAME \
AWSSecretName=$X_GITHUB_SECRET_NAME \
S3TempBucketName=$S3_TEMP_BUCKET_NAME \
GitHubAppID=$X_GITHUB_APP_ID \
GithubAppsClientId=$X_GITHUB_APPS_CLIENT_ID \
Expand Down
10 changes: 5 additions & 5 deletions petercat_utils/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@
| `WEB_URL` | 必选 | 前端 Web 服务的域名 | `https://petercat.ai`
| `STATIC_URL` | 必选 | 静态资源域名 | `https://static.petercat.ai`
| **AWS 相关环境变量** |
| `AWS_GITHUB_SECRET_NAME` | 必选 | AWS 托管的 Github 私钥文件名 | `prod/githubapp/petercat/pem`
| `AWS_STATIC_SECRET_NAME` | 可选 | AWS 托管的 CloudFront 签名私钥名称。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `prod/petercat/static` |
| `AWS_LLM_TOKEN_SECRET_NAME` | 可选 | AWS 托管的 llm 签名私钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm` |
| `AWS_LLM_TOKEN_PUBLIC_NAME` | 可选 | AWS 托管的 llm 签名公钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm/pub` |
| `AWS_STATIC_KEYPAIR_ID` | 可选 | AWS CloudFront 的 Key Pair ID。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `APKxxxxxxxx` |
| `X_GITHUB_SECRET_NAME` | 必选 | AWS 托管的 Github 私钥文件名 | `prod/githubapp/petercat/pem`
| `STATIC_SECRET_NAME` | 可选 | AWS 托管的 CloudFront 签名私钥名称。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `prod/petercat/static` |
| `LLM_TOKEN_SECRET_NAME` | 可选 | AWS 托管的 llm 签名私钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm` |
| `LLM_TOKEN_PUBLIC_NAME` | 可选 | AWS 托管的 llm 签名公钥名称。如果配置了该项,petercat 将使用 RSA 算法托管用户的 LLM Token | `prod/petercat/llm/pub` |
| `STATIC_KEYPAIR_ID` | 可选 | AWS CloudFront 的 Key Pair ID。如果配置了该项,将使用 CloudFront 签名 URL 来保护你的资源。更多信息请参阅 [AWS 文档](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html)。 | `APKxxxxxxxx` |
| `S3_TEMP_BUCKET_NAME` | 可选 | 用于托管 AWS 临时图片文件 S3 的 bucket | `xxx-temp`
| `SQS_QUEUE_URL`| 必选 | AWS SQS 消息队列 URL | `https://sqs.ap-northeast-1.amazonaws.com/xxx/petercat-task-queue`
| **SUPABASE 相关 env** |
Expand Down
10 changes: 5 additions & 5 deletions server/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ AUTH0_CLIENT_SECRET=auth0_client_secret

# OPTIONAL - AWS Configures
SQS_QUEUE_URL=https://sqs.ap-northeast-1.amazonaws.com/{your_aws_user}/{your_aws_sqs_message}
AWS_GITHUB_SECRET_NAME="prod/githubapp/petercat/pem"
AWS_STATIC_SECRET_NAME="prod/petercat/static"
AWS_LLM_TOKEN_SECRET_NAME="prod/petercat/llm"
AWS_LLM_TOKEN_PUBLIC_NAME="petercat/prod/llm/pub"
AWS_STATIC_KEYPAIR_ID="xxxxxx"
X_GITHUB_SECRET_NAME="prod/githubapp/petercat/pem"
STATIC_SECRET_NAME="prod/petercat/static"
LLM_TOKEN_SECRET_NAME="prod/petercat/llm"
LLM_TOKEN_PUBLIC_NAME="petercat/prod/llm/pub"
STATIC_KEYPAIR_ID="xxxxxx"
S3_TEMP_BUCKET_NAME=S3_TEMP_BUCKET_NAME
10 changes: 5 additions & 5 deletions server/.env.local.example
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ AUTH0_CLIENT_SECRET=auth0_client_secret

# OPTIONAL - AWS Configures
SQS_QUEUE_URL=https://sqs.ap-northeast-1.amazonaws.com/{your_aws_user}/{your_aws_sqs_message}
AWS_GITHUB_SECRET_NAME="prod/githubapp/petercat/pem"
AWS_STATIC_SECRET_NAME="prod/petercat/static"
AWS_LLM_TOKEN_SECRET_NAME="prod/petercat/llm"
AWS_LLM_TOKEN_PUBLIC_NAME="petercat/prod/llm/pub"
AWS_STATIC_KEYPAIR_ID="xxxxxx"
X_GITHUB_SECRET_NAME="prod/githubapp/petercat/pem"
STATIC_SECRET_NAME="prod/petercat/static"
LLM_TOKEN_SECRET_NAME="prod/petercat/llm"
LLM_TOKEN_PUBLIC_NAME="petercat/prod/llm/pub"
STATIC_KEYPAIR_ID="xxxxxx"
S3_TEMP_BUCKET_NAME=S3_TEMP_BUCKET_NAME
14 changes: 7 additions & 7 deletions server/aws/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,22 @@
import rsa
from datetime import datetime, timedelta

from utils.get_private_key import get_private_key
from utils.private_key import get_private_key
from .schemas import ImageMetaData
from .constants import S3_TEMP_BUCKET_NAME, STATIC_URL
from .exceptions import UploadError

REGIN_NAME = get_env_variable("AWS_REGION")
AWS_STATIC_SECRET_NAME = get_env_variable("AWS_STATIC_SECRET_NAME")
AWS_STATIC_KEYPAIR_ID = get_env_variable("AWS_STATIC_KEYPAIR_ID")
REGION_NAME = get_env_variable("AWS_REGION")
STATIC_SECRET_NAME = get_env_variable("STATIC_SECRET_NAME")
STATIC_KEYPAIR_ID = get_env_variable("STATIC_KEYPAIR_ID")

def rsa_signer(message):
private_key_str = get_private_key(REGIN_NAME, AWS_STATIC_SECRET_NAME)
private_key_str = get_private_key(STATIC_SECRET_NAME)
private_key = rsa.PrivateKey.load_pkcs1(private_key_str.encode('utf-8'))
return rsa.sign(message, private_key, 'SHA-1')

def create_signed_url(url, expire_minutes=60) -> str:
cloudfront_signer = CloudFrontSigner(AWS_STATIC_KEYPAIR_ID, rsa_signer)
cloudfront_signer = CloudFrontSigner(STATIC_KEYPAIR_ID, rsa_signer)

# 设置过期时间
expire_date = datetime.now() + timedelta(minutes=expire_minutes)
Expand Down Expand Up @@ -65,7 +65,7 @@ def upload_image_to_s3(file, metadata: ImageMetaData, s3_client):
# you need to redirect your static domain to your s3 bucket domain
s3_url = f"{STATIC_URL}/{s3_key}"
signed_url = create_signed_url(url=s3_url, expire_minutes=60) \
if (AWS_STATIC_SECRET_NAME and AWS_STATIC_KEYPAIR_ID) \
if (STATIC_SECRET_NAME and STATIC_KEYPAIR_ID) \
else s3_url
return {"message": "File uploaded successfully", "url": signed_url }
except Exception as e:
Expand Down
3 changes: 1 addition & 2 deletions server/bot/list.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
from typing import Optional

from github import Github
from github import Github, Auth
from core.dao.repositoryConfigDAO import RepositoryConfigDAO
from petercat_utils import get_client
from github import Github, Auth

Check failure on line 6 in server/bot/list.py

View workflow job for this annotation

GitHub Actions / build

Ruff (F811)

bot/list.py:6:20: F811 Redefinition of unused `Github` from line 3

def query_list(
name: Optional[str] = None,
Expand Down
Loading
Loading