Skip to content

Commit

Permalink
feat: update bk-storages to 2.0.0 (#195)
Browse files Browse the repository at this point in the history
  • Loading branch information
narasux authored Oct 24, 2024
1 parent cb23fbb commit 5c0bd5b
Show file tree
Hide file tree
Showing 10 changed files with 154 additions and 247 deletions.
86 changes: 44 additions & 42 deletions .github/workflows/bk-storages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,60 +2,62 @@ name: CI Check for bk-storages

on:
push:
branches: [ master, staging ]
branches: [master, staging]
paths:
- 'sdks/bk-storages/**'
- "sdks/bk-storages/**"
pull_request:
branches: [ master, staging ]
branches: [master, staging]
paths:
- 'sdks/bk-storages/**'
- "sdks/bk-storages/**"
workflow_dispatch:

jobs:
check:
runs-on: macos-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.7
- name: Format with isort
run: |
pip install isort==5.9.2
isort sdks/ --settings-path=sdks/bk-storages/pyproject.toml
- name: Format with black
run: |
pip install black==21.7b0 click==8.0.4
black sdks/ --config=sdks/bk-storages/pyproject.toml
- name: Lint with flake8
run: |
pip install pyproject-flake8
pflake8 sdks/ --config=sdks/bk-storages/pyproject.toml
- name: Lint with mypy
run: |
pip install mypy==0.910 types-requests==2.25.0 types-setuptools==57.0.0 types-dataclasses==0.1.5 types-redis==3.5.4 types-PyMySQL==1.0.0 types-six==0.1.7 types-toml==0.1.3
mypy sdks/bk-storages --config-file=sdks/bk-storages/pyproject.toml
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.11
- name: Format with isort
working-directory: sdks/bk-storages
run: |
pip install isort==5.12.0
isort . --settings-path=pyproject.toml
- name: Format with black
working-directory: sdks/bk-storages
run: |
pip install black==23.7.0 click==8.1.6
black . --config=pyproject.toml
- name: Lint with flake8
working-directory: sdks/bk-storages
run: |
pip install flake8==4.0.1 pyproject-flake8==0.0.1a5
pflake8 . --config=pyproject.toml
- name: Lint with mypy
working-directory: sdks/bk-storages
run: |
pip install mypy==0.910 types-requests==2.31.0.2 types-setuptools==57.4.18 types-dataclasses==0.1.7 types-redis==3.5.18 types-PyMySQL==1.1.0.1 types-six==0.1.9 types-toml==0.1.5
mypy . --config-file=pyproject.toml
test:
strategy:
fail-fast: false
matrix:
python-version: ['3.7', '3.8', '3.9', '3.10']
poetry-version: [1.3.2]
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Set up Poetry
uses: abatilo/actions-poetry@v2.1.0
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Test bk-storages
run: |
cd sdks/bk-storages
poetry install
poetry run pytest
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Set up Poetry
uses: abatilo/actions-poetry@v2.3.0
with:
poetry-version: 1.5.1
- name: Test bk-storages
run: |
cd sdks/bk-storages
poetry install
poetry run pytest
12 changes: 12 additions & 0 deletions sdks/bk-storages/CHANGE.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,29 @@
## Change logs

### 2.0.0

- 支持 Python 3.11,移除对 Python 3.6, 3.7 的支持
- Django 版本要求 >= 3.2.25
- 禁止在保存的文件名中使用绝对路径(如:/etc/passwd)或相对路径(如:../../passwd)

### 1.1.1

- 扩展支持的 Python 版本(>=3.6,<3.12)

### 1.1.0

- 调整 `BKRepoStorage` 公开仓库的 url 实现: 直接返回资源的路径, 而并非生成临时访问地址

### 1.0.8

- 扩展支持的 Python 版本(>=3.6,<3.11), 并修复单元测试在 django>=2.2.27 时无法通过的问题

### 1.0.7

- 修复 `BKRepoStorage` 在保存没有配置名称的 File 对象时会触发无法上传的问题

### 1.0.6

- `BKRepoStorage` 增加 DownloadFailedError 和 UploadFailedError 异常

### 1.0.5
Expand Down
11 changes: 1 addition & 10 deletions sdks/bk-storages/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ storage.delete('/test/temp_file.txt')

## 开发指南

首先安装 [poetry](https://github.com/python-poetry/poetry) ,之后在项目目录下执行 `poetry env use python3.6` 初始化开发用虚拟环境。然后用 `poetry shell` 命令激活虚拟环境。
首先安装 [poetry](https://github.com/python-poetry/poetry) ,之后在项目目录下执行 `poetry env use python3.11` 初始化开发用虚拟环境。然后用 `poetry shell` 命令激活虚拟环境。

- 执行 `poetry install` 安装所有依赖
- 使用 `pytest -s .` 执行所有单元测试
Expand All @@ -174,12 +174,3 @@ storage.delete('/test/temp_file.txt')
### 发布包

首先,执行 `poetry build` 命令在 dist 目录下生成当前版本的包。然后执行 `twine upload dist/* --repository-url {pypi_address} --username {your_name} --password {your_token}` 将其上传到 pypi 服务器上。

### 关于 setup.py

虽然在 [PEP 517](https://python-poetry.org/docs/pyproject/#poetry-and-pep-517) 规范里,Python 包不再需要 `setup.py` 文件。但真正少了 `setup.py` 文件后,会发现有些功能就没法正常使用,比如 pip 的可编辑安装模式、tox 等( [相关文档](https://github.com/python-poetry/poetry/issues/761) )。所以我们仍然需要它。

为了避免维护重复的 `pyproject.toml``setup.py` 文件,我们使用了 [dephell](https://github.com/dephell/dephell) 工具来自动生成 `setup.py` 文件。

- 安装 dephell
- 在根目录执行 `dephell deps convert --from pyproject.toml --to setup.py`
2 changes: 1 addition & 1 deletion sdks/bk-storages/bkstorages/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
"""
__version__ = "1.1.1"
__version__ = "2.0.0"
8 changes: 4 additions & 4 deletions sdks/bk-storages/bkstorages/backends/rgw.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from django.core.files.base import File
from django.core.files.storage import Storage
from django.utils.deconstruct import deconstructible
from django.utils.encoding import filepath_to_uri, force_bytes, force_text, smart_str
from django.utils.encoding import filepath_to_uri, force_bytes, force_str, smart_str
from django.utils.timezone import is_naive, localtime
from six import BytesIO, string_types
from six.moves.urllib import parse as urlparse
Expand All @@ -46,8 +46,8 @@ def percent_encode(input_str, safe=SAFE_CHARS, encoding=None):
encoding = encoding or setting('RGW_FILE_NAME_CHARSET', 'utf-8')

if not isinstance(input_str, string_types):
input_str = force_text(input_str, encoding)
return quote(force_text(input_str, encoding).encode('utf-8'), safe=safe)
input_str = force_str(input_str, encoding)
return quote(force_str(input_str, encoding).encode('utf-8'), safe=safe)


# Start patch botocore to fix Non-ascii filename exception
Expand Down Expand Up @@ -408,7 +408,7 @@ def _encode_name(self, name):
return smart_str(name, encoding=self.file_name_charset)

def _decode_name(self, name):
return force_text(name, encoding=self.file_name_charset)
return force_str(name, encoding=self.file_name_charset)

def _compress_content(self, content):
"""Gzip a given string content."""
Expand Down
6 changes: 3 additions & 3 deletions sdks/bk-storages/bkstorages/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

from django.conf import settings
from django.core.exceptions import ImproperlyConfigured, SuspiciousFileOperation
from django.utils.encoding import force_text
from django.utils.encoding import force_str
from six.moves.urllib.parse import urljoin


Expand Down Expand Up @@ -70,9 +70,9 @@ def safe_join(base, *paths):
Paths outside the base path indicate a possible security
sensitive operation.
"""
base_path = force_text(base)
base_path = force_str(base)
base_path = base_path.rstrip('/')
paths = [force_text(p) for p in paths]
paths = [force_str(p) for p in paths]

final_path = base_path
for path in paths:
Expand Down
Loading

0 comments on commit 5c0bd5b

Please sign in to comment.