使用说明

项目使用 Cookiecutter 编写，用于自动生成预编写好的项目模板。在使用时，通过交互输入参数，控制生成逻辑，以达到一定的自定义化，方便更多场景组合使用。使用项目模板生成项目目录，极大程度减少了在项目开始时搭建项目环境的繁琐步骤，既可以减少编写或复制重复代码，又能避免节省时间。

安装依赖

项目模板使用了新的 PEP-517 打包规范，所以需要使用更新版本的 pip 。

将 pip 升级到最新版本：

pip install -U pip

安装 Cookiecutter :

pip install -U cookiecutter

为了更方便的使用 Cookiecutter ，推荐在系统环境下或者当前用户环境下安装，而不是在虚拟环境中安装。

生成项目模板

在项目生成目录运行如下命令：

cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl

这是使用 Cookiecutter 读取放在 Github 上的项目模板。 Cookiecutter 会首先找到 cookiecutter.json 文件，读取其中的配置，并开启交互提示。

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]:

在使用 Cookiecutter 时，会将项目模板下载到本地作为缓存。如果不是第一次使用，且项目木板有更新时，建议根据提示重新下载项目模板。如果不需要可以跳过，使用上次下载的缓存记录生成项目。

项目名称(project_name)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World

在运行交互时，首先需要输入一个项目名，请输入合法的项目名。

项目名遵循：

只能以 _ 、 a-z 和 A-Z 开头
只能包含 _ 、 a-z 和 A-Z
输入的前后空格会被删除
中间的空格、 - 和 . 都会被替换成 _
所有大写字符都会转换成小写字母

即将输入转换成 Python 的蛇形命名法。

项目包名称(project_slug)

输入的项目名称会在转换和检查后作为合法的 Python 包名称。

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]:

可以直接回车使用自动生成的包名称。或者手动输入想要的包名称。

项目描述(project_description)

输入项目描述文字，此参数暂时没有显示，你可以输入一段描述项目功能的文字。输入时不能换行，如果使用回车，则会直接进入下一个参数的提示。

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.

项目作者(author_name)

author_name [Author]: ming

作者邮箱(author_email)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com

项目版本(version)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com
version [0.1.0]:

输入合法的项目版本号。

PEP-440 中规范了符合 Python 规范的版本号。也可以使用较为通用的语义化版本 2.0 中规定的版本号写法。

Python 版本(python_version)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com
version [0.1.0]:
Select python_version:
1 - 3.10
2 - 3.9
Choose from 1, 2 [1]:

选择使用的 Python 版本。建议使用 Python 3.10 。

使用 SRC 目录结构(use_src_layout)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com
version [0.1.0]:
Select python_version:
1 - 3.10
2 - 3.9
Choose from 1, 2 [1]: 
use_src_layout [y]:

选择是否使用 SRC 目录结构。

如果使用 SRC 目录结构，在项目根目录会有一个 src 的目录， Python 包在 src 下。这种结构方便一个项目包含多个包。

如果不使用 SRC ，则项目的包直接放在项目根目录。

更多关于 Python 目录结构的内容可以阅读项目结构。

使用 Poetry (use_poetry)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com
version [0.1.0]:
Select python_version:
1 - 3.10
2 - 3.9
Choose from 1, 2 [1]: 
use_src_layout [y]: 
use_poetry [y]: 
use_framework[]:
1 - none
2 - pyspark
Choose from 1, 2 [1]:

选择是否使用 Poetry 作为虚拟环境管理工具。默认情况下是推荐使用的， Poetry 是虚拟环境管理工具中的新起之秀，同时包含了打包和发布功能，使用 pyprojrct.toml 管理元数据，减少大量配置文件。

如果不使用，则直接生成一个 requirements.txt 文件，也会包含所需要的依赖。

使用 Docker(use_docker)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]: 
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]: ming
author_email [ming@example.com]: ming@gmail.com
version [0.1.0]: 0.1.0    
Select python_version:
1 - 3.10
2 - 3.9
Choose from 1, 2 [1]: 
use_src_layout [y]: 
use_poetry [y]: 
use_docker [n]:

选择是否使用 Docker 。默认不使用。

如果选择使用，会在项目中生成简单的 Dockerfile 和 .dockerignoer 文件。

CI 工具(ci_tools)

❯ cookiecutter https://github.com/pyloong/cookiecutter-pythonic-project-bigdata-etl
You've downloaded /home/kevin/.cookiecutters/cookiecutter-pythonic-project before. Is it okay to delete and re-download it? [yes]: 
project_name [My Project]: Hello World
project_slug [hello_world]:
project_description [My Awesome Project!]: This is my first python package, i love it.
author_name [Author]:
author_email [author@example.com]:
version [0.1.0]:
Select python_version:
1 - 3.10
2 - 3.9
Choose from 1, 2 [1]:
use_src_layout [y]:
use_poetry [y]:
use_docker [n]:
Select ci_tools:
1 - none
2 - Gitlab
3 - Github
Choose from 1, 2, 3 [1]:

选择使用的 CI 工具，可选的有 Gitlab 和 Github 。如果选择一个 CI 工具，默认会生成项目的测试、构建、发布到索引服务器的三步操作。

使用生成后的项目

当上面的所有步骤操作完成后，就会在当前目录生成一个 Python 项目，其目录结构如下：


hello_world
│ .gitignore
│ LICENSE
│ pyproject.toml
│ README.md
│ tox.ini
│
├─src
│  │  __init__.py
│  │
│  └─pyspark_etl_template
│     │  cmdline.py
│     │  executor.py
│     │  constants.py
│     │  __init__.py
│     │
│     ├─configs
│     │  │  dev.toml
│     │  │  global.toml
│     │  │  prod.toml
│     │  └─ test.toml
│     │
│     ├─dependecies
│     │  │  log.py
│     │  │  config.py
│     │  │  spark.py
│     │  └─ __init__.py
│     │
│     ├─tasks
│     │  │  __init__.py
│     │  │
│     │  └─abstract
│     │     │  task.py
│     │     │  transform.py
│     │     └─ __init__.py
│     │
│     └─utils
│        │  exception.py
│        └─ __init__.py
|
├─tests
|   │  conftest.py
|   │  test_version.py
|   └─ __init__.py
|   
└─docs
    └─ development.md

在项目根目录，包含了一些描述性的文件：

LICENSE ：项目的许可证
pyproject.toml ：由 Poetry 生成的带有 Poetry 配置的项目描述文件。
README.md ：项目自述文件
tox.ini ：自动化测试工具 Tox 的配置文件。

还有一些目录：

docs ：文档目录，用来记录开发和使用时的文档
src ：使用了 SRC 目录结构生成目录，用来放置项目的包
tests ：测试目录，用来放置对 src 下面的包做测试的目录。

使用项目

项目模板生成后，默认会在的当前目录生成一个可用的 Python 项目。

Python 项目开发时，强烈建议使用虚拟环境管理 Python 项目的依赖。在前面生成项目模板是，会默认使用 Poetry 作为虚拟环境管理工具。当然你可以不使用 Poetry ，选择更常见的 Virtualenv 。

后续步骤都假设你已经了解 Poetry 并使用 Poetry 。

如果没有安装 Poetry ，可以使用 pip 命令安装。为了方便使用，推荐在系统环境下或者当前用户环境下安装。

## 升级 pip
pip install -U pip
pip install -U poetry

## 进入项目目录
cd  hello_world

## 创建虚拟环境，并安装依赖
poetry install

## 进入虚拟环境
poetry shell

为了减少开发过程中遇到模板本身错误导致开发异常的情况，在开发前首先运行一遍自动化测试，直接执行 tox 命令。

.package create: D:\GitHub\hello_world\.tox\.package
.package installdeps: poetry-core>=1.0.0
py310 create: D:\GitHub\hello_world\.tox\py310
py310 installdeps: poetry
py310 inst: D:\GitHub\hello_world\.tox\.tmp\package\1\hello_world-0.1.0.tar.gz
py310 installed: attrs==22.1.0,CacheControl==0.12.11,cachy==0.3.0,certifi==2022.9.14,charset-normalizer==2.1.1,cleo==1.0.0a5,click==8.1.3,colorama==0.4.5,crashtest==0.3.1,distlib==0.3.6,dulwich==0.20.46,dynaconf==3.1.9,filelock==3.8.0,hello-world @ file:///D:/GitHub/hello_world/.tox/.tmp/package/1/hello_world-0.1.0.tar.gz,html5lib==1.1,idna==3.4,jaraco.classes==3.2.2,jsonschema==4.16.0,keyring==23.9.1,lockfile==0.12.2,more-itertools==8.14.0,msgpack==1.0.4,packaging==21.3,pbr==5.10.0,pexpect==4.8.0,pkginfo==1.8.3,platformdirs==2.5.2,poetry==1.2.0,poetry-core==1.1.0,poetry-plugin-export==1.0.6,ptyprocess==0.7.0,py4j==0.10.9.5,pylev==1.4.0,pyparsing==3.0.9,pyrsistent==0.18.1,pyspark==3.3.0,pywin32-ctypes==0.2.0,requests==2.28.1,requests-toolbelt==0.9.1,shellingham==1.5.0,six==1.16.0,stevedore==4.0.0,tomlkit==0.11.4,urllib3==1.26.12,virtualenv==20.16.5,webencodings==0.5.1
py310 run-test-pre: PYTHONHASHSEED='461'
py310 run-test: commands[0] | poetry install -v
Using virtualenv: D:\GitHub\hello_world\.tox\py310
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 28 installs, 0 updates, 0 removals, 13 skipped

  • Installing markupsafe (2.1.1)
  • Installing python-dateutil (2.8.2)
  • Installing pyyaml (6.0)
  • Installing zipp (3.8.1)
  • Installing ghp-import (2.1.0)
  • Installing importlib-metadata (4.12.0)
  • Installing jinja2 (3.1.2)
  • Installing mergedeep (1.3.4)
  • Installing markdown (3.3.7)
  • Installing watchdog (2.1.9)
  • Installing wrapt (1.14.1)
  • Installing pyyaml-env-tag (0.1)
  • Installing lazy-object-proxy (1.7.1)
  • Installing astroid (2.12.9)
  • Installing dill (0.3.5.1)
  • Installing iniconfig (1.1.1)
  • Installing mkdocs (1.3.1)
  • Installing isort (5.10.1)
  • Installing pluggy (1.0.0)
  • Installing pymdown-extensions (9.5)
  • Installing mccabe (0.7.0)
  • Installing mkdocs-material-extensions (1.0.3)
  • Installing pygments (2.13.0)
  • Installing tomli (2.0.1)
  • Installing py (1.11.0)
  • Installing attrs (22.1.0): Skipped for the following reason: Already installed
  • Installing click (8.1.3): Skipped for the following reason: Already installed
  • Installing colorama (0.4.5): Skipped for the following reason: Already installed
  • Installing dynaconf (3.1.9): Skipped for the following reason: Already installed
  • Installing mkdocs-material (8.4.4)
  • Installing packaging (21.3): Skipped for the following reason: Already installed
  • Installing pytest (7.1.3)
  • Installing platformdirs (2.5.2): Skipped for the following reason: Already installed
  • Installing py4j (0.10.9.5): Skipped for the following reason: Already installed
  • Installing pyspark (3.3.0): Skipped for the following reason: Already installed
  • Installing stevedore (4.0.0): Skipped for the following reason: Already installed
  • Installing pyparsing (3.0.9): Skipped for the following reason: Already installed
  • Installing tomlkit (0.11.4): Skipped for the following reason: Already installed
  • Installing six (1.16.0): Skipped for the following reason: Already installed
  • Installing pylint (2.15.2)
  • Installing pbr (5.10.0): Skipped for the following reason: Already installed

Installing the current project: hello_world (0.1.0)
py310 run-test: commands[1] | poetry run pytest tests
================================================= test session starts =================================================
platform win32 -- Python 3.10.5, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox\py310\.pytest_cache
rootdir: D:\GitHub\hello_world
collected 1 item

tests\test_version.py .                                                                                          [100%]

================================================== 1 passed in 0.01s ==================================================
isort create: D:\GitHub\hello_world\.tox\isort
isort installdeps: isort
isort inst: D:\GitHub\hello_world\.tox\.tmp\package\1\hello_world-0.1.0.tar.gz
isort installed: click==8.1.3,colorama==0.4.5,dynaconf==3.1.9,hello-world @ file:///D:/GitHub/hello_world/.tox/.tmp/package/1/hello_world-0.1.0.tar.gz,isort==5.10.1,pbr==5.10.0,py4j==0.10.9.5,pyspark==3.3.0,stevedore==4.0.0
isort run-test-pre: PYTHONHASHSEED='461'
isort run-test: commands[0] | isort . --check-only --diff
Skipped 1 files
pylint create: D:\GitHub\hello_world\.tox\pylint
pylint installdeps: poetry
pylint inst: D:\GitHub\hello_world\.tox\.tmp\package\1\hello_world-0.1.0.tar.gz
pylint installed: attrs==22.1.0,CacheControl==0.12.11,cachy==0.3.0,certifi==2022.9.14,charset-normalizer==2.1.1,cleo==1.0.0a5,click==8.1.3,colorama==0.4.5,crashtest==0.3.1,distlib==0.3.6,dulwich==0.20.46,dynaconf==3.1.9,filelock==3.8.0,hello-world @ file:///D:/GitHub/hello_world/.tox/.tmp/package/1/hello_world-0.1.0.tar.gz,html5lib==1.1,idna==3.4,jaraco.classes==3.2.2,jsonschema==4.16.0,keyring==23.9.1,lockfile==0.12.2,more-itertools==8.14.0,msgpack==1.0.4,packaging==21.3,pbr==5.10.0,pexpect==4.8.0,pkginfo==1.8.3,platformdirs==2.5.2,poetry==1.2.0,poetry-core==1.1.0,poetry-plugin-export==1.0.6,ptyprocess==0.7.0,py4j==0.10.9.5,pylev==1.4.0,pyparsing==3.0.9,pyrsistent==0.18.1,pyspark==3.3.0,pywin32-ctypes==0.2.0,requests==2.28.1,requests-toolbelt==0.9.1,shellingham==1.5.0,six==1.16.0,stevedore==4.0.0,tomlkit==0.11.4,urllib3==1.26.12,virtualenv==20.16.5,webencodings==0.5.1
pylint run-test-pre: PYTHONHASHSEED='461'
pylint run-test: commands[0] | poetry install -v
Using virtualenv: D:\GitHub\hello_world\.tox\pylint
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 28 installs, 0 updates, 0 removals, 13 skipped

  • Installing markupsafe (2.1.1)
  • Installing python-dateutil (2.8.2)
  • Installing pyyaml (6.0)
  • Installing zipp (3.8.1)
  • Installing ghp-import (2.1.0)
  • Installing importlib-metadata (4.12.0)
  • Installing jinja2 (3.1.2)
  • Installing mergedeep (1.3.4)
  • Installing pyyaml-env-tag (0.1)
  • Installing lazy-object-proxy (1.7.1)
  • Installing watchdog (2.1.9)
  • Installing wrapt (1.14.1)
  • Installing markdown (3.3.7)
  • Installing astroid (2.12.9)
  • Installing dill (0.3.5.1)
  • Installing isort (5.10.1)
  • Installing mkdocs (1.3.1)
  • Installing pluggy (1.0.0)
  • Installing pygments (2.13.0)
  • Installing tomli (2.0.1)
  • Installing mccabe (0.7.0)
  • Installing mkdocs-material-extensions (1.0.3)
  • Installing iniconfig (1.1.1)
  • Installing pymdown-extensions (9.5)
  • Installing py (1.11.0)
  • Installing attrs (22.1.0): Skipped for the following reason: Already installed
  • Installing click (8.1.3): Skipped for the following reason: Already installed
  • Installing colorama (0.4.5): Skipped for the following reason: Already installed
  • Installing dynaconf (3.1.9): Skipped for the following reason: Already installed
  • Installing py4j (0.10.9.5): Skipped for the following reason: Already installed
  • Installing pyparsing (3.0.9): Skipped for the following reason: Already installed
  • Installing pytest (7.1.3)
  • Installing stevedore (4.0.0): Skipped for the following reason: Already installed
  • Installing mkdocs-material (8.4.4)
  • Installing pyspark (3.3.0): Skipped for the following reason: Already installed
  • Installing platformdirs (2.5.2): Skipped for the following reason: Already installed
  • Installing packaging (21.3): Skipped for the following reason: Already installed
  • Installing tomlkit (0.11.4): Skipped for the following reason: Already installed
  • Installing six (1.16.0): Skipped for the following reason: Already installed
  • Installing pylint (2.15.2)
  • Installing pbr (5.10.0): Skipped for the following reason: Already installed

Installing the current project: hello_world (0.1.0)
pylint run-test: commands[1] | poetry run pylint tests src

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 8.37/10, +1.63)

_______________________________________________________ summary _______________________________________________________
  py310: commands succeeded
  isort: commands succeeded
  pylint: commands succeeded
  congratulations :)

可以看到 tox 做了如下事情：

执行了 py310 的测试，
isort 的包导入检查，
pylint 的 Python 语法规范检查。

当所有检查都正常时， Tox 的自动化逻辑才会正常通过，否则会报出异常，此时你需要根据提示调整代码，并再次运行，直到全部通过。

如果你讨厌这些繁琐的检查，也可以修改 tox.ini 文件，删除不喜欢的配置，或者不运行 tox 。但是并不推荐你这么做。因为良好的编码习惯是一个优秀的开发人员最基本的要求。

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

usage.md

usage.md

使用说明

安装依赖

生成项目模板

项目名称(project_name)

项目包名称(project_slug)

项目描述(project_description)

项目作者(author_name)

作者邮箱(author_email)

项目版本(version)

Python 版本(python_version)

使用 SRC 目录结构(use_src_layout)

使用 Poetry (use_poetry)

使用 Docker(use_docker)

CI 工具(ci_tools)

使用生成后的项目

使用项目

Files

usage.md

Latest commit

History

usage.md

File metadata and controls

使用说明

安装依赖

生成项目模板

项目名称(project_name)

项目包名称(project_slug)

项目描述(project_description)

项目作者(author_name)

作者邮箱(author_email)

项目版本(version)

Python 版本(python_version)

使用 SRC 目录结构(use_src_layout)

使用 Poetry (use_poetry)

使用 Docker(use_docker)

CI 工具(ci_tools)

使用生成后的项目

使用项目