Feat/build with poetry (#132)

* versions frozen * example poetry * typos fixed * poetry scripts * all scripts ready * github actions updated * linting fixed * remove get_extra_deps_line_number * add inplace patching for sphinx This requires the `wrapt` library which I haven't specified yet in project requirements. * codestyle * add dependencies required for poetry Added wrapt dependency as a core dependency Added urllib3 dependency to core (poetry requires urllib3<2.0 to function but some extra libraries update its version) Maybe should use a different approach for these dependencies (not install them in core) * fix coverage job * separate doc and test; add test_no_cov * change called script in jobs; modify dff installation - Add pip upgrade for no reason - Install specific extras instead of all - Add flags to fix problems with poetry install - Temporary fix of coverage job (poetry run test doesn't work) * dependencies update * lint applied * tests + docs fixed * versions updated * scripting fixed * actions updated * attempt to fix test_full * docker client adapted * docker context manager once again updated * tests no deps fixed * coverage test fixed * setuptools added * test coverage poetry install added * all dependencies added * test coverage now in main * backup files fixed + build and upload fail added * context manager replaced with decorator (better) * 1 return added * add patch to suppress logger * new dependencies * all extras * docker quiet + containers always down * specified exception * relint * reformatted * poetry scripts return ints * benchmark added to coverage * test full no coverage * image building postponed * otel volumes path fixed * image build placing reverted * building removed * lint removed from coverage * tests deps fixed * poetry tests docker added * tqdm added to benchmark * iploading fixed * pypi token added * scripts updated * lint + dry build * build with publish + dirs exist ok * apt -> apt-get + glob -> pathlib * fix bumpversion * contributing changed * some reviews addressed * pyproject updated to 0.6.1 * lint applied * docker compose call updated * test_ch exception check * test tutorials exception check * exception in rows * assert reverted * some comments fixed * restricted, docker tests run on windows * docker any platform reverted * versions commenting started * codestyle comments added * extras -> groups * lint applied * comment added * poethepoet introduced * poe added as lib in poetry * default script type changed * codestyle fixed * extract dff version from importlib * add emails; merge maintainers with authors * rework codestyle && fix codestyle * remove docker-compose usage * fix test_coverage && update includes and excludes Instead of removing all directories to check that tests pass and do not miss any json files in `dff` it would be better to simply include all files under `dff` into source package. Also remove redundant excludes and includes. * remove lint from tests These are separate tasks. * replace returns with exits in scripts * move coverage config to pyproject.toml * freeze coverage version Coverage 7.3.3 breaks `pragma: no cover` -- https://www.github.com/nedbat/coveragepy/issues/1713 * add poetry lock to git The error with test_coverage was caused by new `coverage` version breaking things. Keeping poetry lock in git would allow to separate errors caused by our code from those caused by dependency updates: https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control * update poetry lock && add information about poetry lock updates * update ignore files * make devel group required All devel processes require it. * update dff version * remove redundant groups from workflows * keep poetry.lock during cleaning * unfreeze and update devel deps Poetry lock allows us to freeze dependencies without doing so in pyproject.toml. * freeze urllib only in devel urllib breaks poetry, so we want to restrict its version only in devel environments * replace exclude_lines with exclude_also * add python -m for workflows commands --------- Co-authored-by: Roman Zlobin <[email protected]>
deeppavlov · Dec 20, 2023 · 8dc9b1b · 8dc9b1b
1 parent 677ee7a
commit 8dc9b1b
Show file tree

Hide file tree

Showing 49 changed files with 7,598 additions and 703 deletions.
diff --git a/.coveragerc b/.coveragerc
diff --git a/.dockerignore b/.dockerignore
@@ -1,7 +1,6 @@
 *.DS_Store*
 *.egg-info/
 dist/
-venv/
 build/
 docs/source/apiref
 docs/source/_misc
@@ -22,9 +21,9 @@ GlobalUserTableAccessor*
 memory_debugging*
 opening_database*
 _globals.py
-venv*
 .vscode
 .coverage
+.coverage.*
 .pytest_cache
 htmlcov
 tutorials/context_storages/dbs

diff --git a/.github/process_github_events.py b/.github/process_github_events.py
@@ -46,6 +46,7 @@ def post_comment_on_pr(comment: str, pr_number: int):
 Here's a release checklist:
 
 - [ ] Update package version
+- [ ] Update `poetry.lock`
 - [ ] Change PR merge option
 - [ ] Test modules without automated testing:
   - [ ] Requiring telegram `api_id` and `api_hash`

diff --git a/.github/workflows/build_and_publish_docs.yml b/.github/workflows/build_and_publish_docs.yml
@@ -27,17 +27,14 @@ jobs:
         with:
           python-version: 3.9
 
-      - name: Build images
-        run: |
-            make docker_up
-
       - uses: r-lib/actions/setup-pandoc@v2
         with:
           pandoc-version: '3.1.6'
 
-      - name: install dependencies
+      - name: setup poetry and install dependencies
         run: |
-            make venv
+          python -m pip install --upgrade pip poetry
+          python -m poetry install --with tutorials,docs --all-extras --no-ansi --no-interaction
 
       - name: build documentation
         env:
@@ -46,7 +43,7 @@ jobs:
           TG_API_HASH: ${{ secrets.TG_API_HASH }}
           TG_BOT_USERNAME: ${{ secrets.TG_BOT_USERNAME }}
         run: |
-            make doc
+          python -m poetry run poe docs
 
       - name: remove jekyll theming
         run: touch docs/build/.nojekyll

diff --git a/.github/workflows/build_and_upload_release.yml b/.github/workflows/build_and_upload_release.yml
@@ -1,9 +1,13 @@
 name: build_and_upload_release
 
 on:
-  push:
-    tags:
-      - '*'
+  release:
+    types:
+      - published
+  pull_request:
+    branches:
+      - dev
+      - master
 
 jobs:
   build:
@@ -16,22 +20,24 @@ jobs:
       - name: set up python
         uses: actions/setup-python@v4
 
-      - name: install latest pip, setuptools, twine + wheel
+      - name: setup poetry
         run: |
-          python -m pip install --upgrade pip setuptools twine wheel
+          python -m pip install --upgrade pip poetry
 
-      - name: build wheels
+      - name: build wheels and test uploading to pypi
+        if: startsWith(github.ref, 'refs/tags/v') != true
         run: |
-          python setup.py bdist_wheel
-          python setup.py sdist
+          python -m poetry --build publish --dry-run
 
-      - name: upload to pypi via twine
+      - name: build wheels and upload to pypi
+        if: startsWith(github.ref, 'refs/tags/v')
         env:
-          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+          POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_TOKEN }}
         run: |
-          twine upload --verbose -u '__token__' dist/*
+          python -m poetry --build publish
 
       - name: upload binaries into release
+        if: startsWith(github.ref, 'refs/tags/v')
         uses: svenstaro/upload-release-action@v2
         with:
           repo_token: ${{ secrets.GITHUB_TOKEN }}

diff --git a/.github/workflows/codestyle.yml b/.github/workflows/codestyle.yml
@@ -26,10 +26,11 @@ jobs:
         with:
           python-version: 3.9
 
-      - name: install dependencies
+      - name: setup poetry and install dependencies
         run: |
-            make venv
+          python -m pip install --upgrade pip poetry
+          python -m poetry install --with lint --no-ansi --no-interaction
 
       - name: run codestyle
         run: |
-            make lint
+          python -m poetry run poe lint
diff --git a/.github/workflows/test_coverage.yml b/.github/workflows/test_coverage.yml
@@ -25,30 +25,15 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Build images
-        run: |
-          make docker_up
-
       - name: set up python 3.10
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
 
-      - name:  build & install package
-        run: |
-          python -m pip install build
-          python -m build
-          find . -name "*.whl" -exec python -m pip install {}[test_full] ';'
-
-      - name: clean environment
+      - name: setup poetry and install dependencies
         run: |
-          export backup_files=( tests tutorials utils .env_file makefile .coveragerc pytest.ini docs )
-          mkdir /tmp/backup
-          for i in "${backup_files[@]}" ; do mv "$i"  /tmp/backup ; done
-          rm -rf ..?* .[!.]* *
-          for i in "${backup_files[@]}" ; do mv  "/tmp/backup/$i"  .  ; done
-          rm -rf /tmp/backup
-          touch venv  # disable venv target
+          python -m pip install --upgrade pip poetry
+          python -m poetry install --with test,tutorials --all-extras --no-ansi --no-interaction
 
       - name: run tests
         env:
@@ -57,4 +42,4 @@ jobs:
           TG_API_HASH: ${{ secrets.TG_API_HASH }}
           TG_BOT_USERNAME: ${{ secrets.TG_BOT_USERNAME }}
         run: |
-          make test TEST_ALLOW_SKIP=telegram
+          python -m poetry run poe test_all
diff --git a/.github/workflows/test_full.yml b/.github/workflows/test_full.yml
@@ -27,21 +27,15 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Build images
-        if: matrix.os == 'ubuntu-latest'
-        run: |
-          make docker_up
-
       - name: set up python ${{ matrix.python-version }}
         uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: install dependencies
         run: |
-          python -m pip install --upgrade pip
-          python -m pip install -e .[test_full]
-        shell: bash
+          python -m pip install --upgrade pip poetry
+          python -m poetry install --with test,tutorials --all-extras --no-ansi --no-interaction
 
       - name: run pytest
         env:
@@ -50,32 +44,22 @@ jobs:
           TG_API_HASH: ${{ secrets.TG_API_HASH }}
           TG_BOT_USERNAME: ${{ secrets.TG_BOT_USERNAME }}
         run: |
-          if [ "$RUNNER_OS" == "Linux" ]; then
-              source <(cat .env_file | sed 's/=/=/' | sed 's/^/export /')
-              pytest --tb=long -vv --cache-clear --no-cov --allow-skip=telegram tests/
-          else
-              pytest -m "not docker" --tb=long -vv --cache-clear --no-cov --allow-skip=telegram,docker tests/
-          fi
-        shell: bash
+          python -m poetry run poe test_no_cov
+
   test_no_deps:
     runs-on: "ubuntu-latest"
     steps:
       - uses: actions/checkout@v4
 
-      - name: Build images
-        run: |
-          make docker_up
-
       - name: set up python 3.8
         uses: actions/setup-python@v4
         with:
           python-version: 3.8
 
       - name: install dependencies
         run: |
-          python -m pip install --upgrade pip
-          python -m pip install -e .[tests]
-        shell: bash
+          python -m pip install --upgrade pip poetry
+          python -m poetry install --with test --no-ansi --no-interaction
 
       - name: run pytest
         env:
@@ -84,6 +68,4 @@ jobs:
           TG_API_HASH: ${{ secrets.TG_API_HASH }}
           TG_BOT_USERNAME: ${{ secrets.TG_BOT_USERNAME }}
         run: |
-          source <(cat .env_file | sed 's/=/=/' | sed 's/^/export /')
-          pytest --tb=long -vv --cache-clear --no-cov --allow-skip=all tests/
-        shell: bash
+          python -m poetry run poe test_no_deps
diff --git a/.gitignore b/.gitignore
@@ -1,7 +1,6 @@
 *.DS_Store*
 *.egg-info/
 dist/
-venv/
 build/
 docs/source/apiref
 docs/source/_misc
@@ -22,9 +21,9 @@ GlobalUserTableAccessor*
 memory_debugging*
 opening_database*
 _globals.py
-venv*
 .vscode
 .coverage
+.coverage.*
 .pytest_cache
 htmlcov
 tutorials/context_storages/dbs