Merge branch 'release/v0.4.0'

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.4
+current_version = 0.4.0
 commit = False
 tag = False
 allow_dirty = False

.github/workflows/tox.yaml

@@ -22,16 +22,19 @@ jobs:
     services:
       remote-storage:
         image: bitnami/minio:latest
-        env:
-          MINIO_ACCESS_KEY: minio
-          MINIO_SECRET_KEY: minio123
         ports:
-        - 9000:9000
+        - "9000:9000"
+        - "9001:9001"
+        env:
+          MINIO_ROOT_USER: minio-root-user
+          MINIO_ROOT_PASSWORD: minio-root-password
 
     steps:
       # pandoc needed for docu, see https://nbsphinx.readthedocs.io/en/0.7.1/installation.html?highlight=pandoc#pandoc
       - name: Install Non-Python Packages
         run: apt-get update -yq && apt-get -yq install pandoc git git-lfs rsync
+      - name: Safe directory workaround
+        run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
       - uses: actions/[email protected]
         with:
           fetch-depth: 0
@@ -43,6 +46,8 @@ jobs:
       - name: Merge develop into current branch
         if: github.ref != 'refs/heads/develop'
         run: |
+          git config --global user.email "[email protected]"
+          git config --global user.name "github_robot_user"
           git fetch origin develop:develop --update-head-ok
           git merge develop
       - name: Setup Python 3.8

.gitlab-ci.yml

@@ -42,8 +42,8 @@ tox_recreate:
     - name: bitnami/minio:latest
       alias: remote-storage
   variables:
-    MINIO_ACCESS_KEY: minio
-    MINIO_SECRET_KEY: minio123
+    MINIO_ACCESS_KEY: minio-root-user
+    MINIO_SECRET_KEY: minio-root-password
   script:
     - pip install tox
     - tox -r
@@ -67,8 +67,8 @@ tox_use_cache:
     - name: bitnami/minio:latest
       alias: remote-storage
   variables:
-    MINIO_ACCESS_KEY: minio
-    MINIO_SECRET_KEY: minio123
+    MINIO_ACCESS_KEY: minio-root-user
+    MINIO_SECRET_KEY: minio-root-password
   script:
     - pip install tox
     - tox

.pre-commit-config.yaml

@@ -1,11 +1,11 @@
 repos:
   - repo: https://github.com/psf/black
-    rev: 20.8b1
+    rev: 23.1.0
     hooks:
       - id: black
         language_version: python3
   - repo: https://github.com/PyCQA/isort
-    rev: 5.6.4
+    rev: 5.12.0
     hooks:
       - id: isort
   - repo: https://github.com/kynan/nbstripout

README_dev.md → CONTRIBUTING.md

@@ -1,12 +1,19 @@
 accsr library
 =============
 
-This repository contains the accsr python library together with utilities for building, testing, 
-documentation and configuration management. The library can be installed locally by running
+## Installing
 
-```python setup.py install```
+You should install the library together with all dependencies as an editable package. We strongly suggest to use some form of virtual environment for working with the library. E.g. with conda:
 
-from the root directory.
+```shell
+conda create -n accsr python=3.8
+conda activate accsr
+pip install -e .
+pip install -r requirements-dev.txt -r requirements-docs.txt -r requirements-test.txt -r requirements-linting.txt -r requirements-coverage.txt
+```
+
+from the root directory. Strictly speaking, you wouldn't
+need to install the dev dependencies, as they are installed by `tox` on the file, but they are useful for development without using tox.
 
 This project uses the [black](https://github.com/psf/black) source code formatter
 and [pre-commit](https://pre-commit.com/) to invoke it as a Git pre-commit hook.
@@ -28,14 +35,15 @@ and documentation.
 Before pushing your changes to the remote it is often useful to execute `tox` locally in order to
 detect mistakes early on.
 
-We strongly suggest to use some form of virtual environment for working with the library. E.g. with conda:
-```shell script
-conda create -n accsr python=3.8
-conda activate accsr
-pip install -r requirements.txt -r requirements-dev.txt
-```
 
 ### Testing and packaging
+
+Local testing is done with pytest and tox. Note that you can 
+perform each part of the test pipeline individually as well, either
+by using `tox -e <env>` or by executing the scripts in the
+`build_scripts` directory directly. See the `tox.ini` file for the
+list of available environments and scripts.
+
 The library is built with tox which will build and install the package and run the test suite.
 Running tox will also generate coverage and pylint reports in html and badges. 
 You can configure pytest, coverage and pylint by adjusting [pytest.ini](pytest.ini), [.coveragerc](.coveragerc) and
@@ -65,40 +73,11 @@ tests. Have a look at the example notebook for an explanation of how this works.
 You might wonder why the requirements.txt already contains numpy. The reason is that tox seems to have a problem with empty
 requirements files. Feel free to remove numpy once you have non-trivial requirements
 
-## Configuration Management
-The repository also includes configuration utilities that are often helpful when using data-related libraries. 
-They do not form part of the resulting package, you can (and probably should) adjust them to your needs.
-
-## CI/CD and Release Process
-This repository contains a gitlab ci/cd pipeline that will run the test suite and
-publish docu, badges and reports. Badges can accessed from the pipeline's artifacts, e.g. for the coverage badge
-the url will be:
-```
-https://gitlab.aai.lab/%{project_path}/-/jobs/artifacts/develop/raw/badges/coverage.svg?job=tox
-```
-
-### Development and Release Process
-
-In order to be able to automatically release new versions of the package from develop and master, the
- CI pipeline should have access to the following variables (they should already be set on global level):
-
-```
-PYPI_REPO_URL
-PYPI_REPO_USER
-PYPI_REPO_PASS
-```
-
-They will be used in the release steps in the gitlab pipeline.
-
-You will also need to set up Gitlab CI deploy keys for 
-automatically committing from the develop pipeline during version bumping
-
 
 #### Automatic release process
 
 In order to create an automatic release, a few prerequisites need to be satisfied:
 
-- The project's virtualenv needs to be active
 - The repository needs to be on the `develop` branch
 - The repository must be clean (including no untracked files)
 

README.md

@@ -1,14 +1,16 @@
-# accsr 
+# accsr: Simple tools for loading data and configurations 
 
 This lightweight library contains utilities for managing, loading, uploading, opening and generally wrangling data and
 configurations. It was battle tested in multiple projects at appliedAI. 
 
-Please open new issues for bugs, feature requests and extensions. See more details about the structure and
-workflow in the [developer's readme](README_dev.md). 
+The main useful abstractions provided by this library are:
+1. The `RemoteStorage`
+class for a git-like, programmatic access to data stored in any cloud storage.
+2. The configuration module for a simple, yet powerful configuration management.
 
 ## Overview
 
-Source code documentation and usage examples are [here](TODO). We also provide notebooks with examples in *TODO*.
+Source code documentation and usage examples are [here](https://appliedai-initiative.github.io/accsr/docs/).
 
 ## Installation
 
@@ -22,3 +24,9 @@ To live on the edge, install the latest develop version with
 pip install --pre accsr
 ```
 
+## Contributing
+
+Please open new issues for bugs, feature requests and extensions. See more details about the structure and
+workflow in the [developer's readme](README_dev.md). The coverage and pylint report can be found on the project's
+[github pages](https://appliedai-initiative.github.io/accsr/).
+