Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Screenshots with rich-codex #22

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
09240d9
Add bug report template
apeltzer Sep 18, 2024
d2be29f
Add more templates
apeltzer Sep 18, 2024
3b01a5a
Adding deploy to pypi
apeltzer Sep 18, 2024
e61dff2
First stab at running CI tests
apeltzer Sep 18, 2024
ca19436
Unit tests only for Python 3.13
apeltzer Sep 18, 2024
beac216
Fix libgit
apeltzer Sep 18, 2024
eb3ce01
Whitespace
apeltzer Sep 18, 2024
b410741
Should work now
apeltzer Sep 18, 2024
91d3969
Explain why we need this
apeltzer Sep 18, 2024
7dbf0f4
Stick to python 3.12 for now
apeltzer Sep 18, 2024
05a3775
Add codecov, improve pyproject toml
apeltzer Sep 18, 2024
1e4faba
Install test environment
apeltzer Sep 18, 2024
385ba30
Improve readme with autogenerated screenshots :)
apeltzer Sep 18, 2024
bd47427
Add more screenshots
apeltzer Sep 18, 2024
1339de0
Adding in quarto pip tests requirements
apeltzer Sep 18, 2024
25f4585
Quarto via pip
apeltzer Sep 18, 2024
76f8dc4
Add pandocfilters
apeltzer Sep 18, 2024
cf1978c
Move towards trusted publishing
apeltzer Sep 19, 2024
ed10622
Autofixing
apeltzer Sep 19, 2024
4f195b4
Panflutes
apeltzer Sep 19, 2024
b092541
Typo
apeltzer Sep 19, 2024
a720591
Flute, not flutes
apeltzer Sep 19, 2024
18a7cd5
Not necessary
apeltzer Sep 19, 2024
1dd73a2
Use quarto GH action to setup quarto
apeltzer Sep 19, 2024
b38ea7b
Merge branch 'main' into add-ci
grst Sep 21, 2024
aee6d9c
Don't enforce quarto version
grst Sep 21, 2024
b867c3c
Fix v8 options
grst Sep 21, 2024
c53e878
update release workflow
grst Sep 21, 2024
9483862
Update pre-commit
grst Sep 21, 2024
3648619
autoformat
grst Sep 21, 2024
58b55d5
Merge remote-tracking branch 'origin/main' into add-ci
grst Sep 21, 2024
35d7ca7
run rich codex automatically
grst Sep 21, 2024
793f8ff
disable codecov for now; can revisit later
grst Sep 21, 2024
02f7e66
attempt fix rich codex autocommits
grst Sep 21, 2024
6420750
Pre-commit
grst Sep 21, 2024
7420cf8
Try fix rich
grst Sep 21, 2024
83801ec
test pre-commit action
grst Sep 21, 2024
cdf838a
Generate new screengrabs with rich-codex
github-actions[bot] Sep 21, 2024
08b6192
Generate new screengrabs with rich-codex
github-actions[bot] Sep 21, 2024
c05d01a
Merge branch 'main' into rich-codex
apeltzer Sep 23, 2024
212196f
Apply automatic changes
apeltzer Sep 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/screenshots.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Docs screenshots
on:
workflow_dispatch:
pull_request:

jobs:
rich_codex:
runs-on: ubuntu-latest
steps:
- name: Check out DSO code
uses: actions/checkout@v4
with:
ref: ${{ github.head_ref }}

- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"

- name: Install DSO
run: pip install .

- name: Generate terminal images with rich-codex
uses: ewels/rich-codex@v1
with:
clean_img_paths: docs/images/screenshots/*.svg
commit_changes: false
skip_git_checks: true

- uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: Rich codex
file_pattern: "docs/*.png"
44 changes: 12 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DSO: data science operations

<img src="img/dso_kraken.jpg" alt="DSO Kraken" width="250" />
<img src="docs/img/dso_kraken.jpg" alt="DSO Kraken" width="250" />

*DSO* is a command line helper for building reproducible data anlaysis projects with ease.
It builds on top of [dvc](https://github.com/iterative/dvc) for data versioning and provides project
Expand All @@ -21,22 +21,14 @@ step of your analysis (usually one script with defined inputs and outputs) organ
cannot be nested. A *folder* is used to organize stages in a hierarchical way within the project.

You can use `dso init` to create a new project
```
$> dso init
Please enter the name of the project, e.g. "single_cell_lung_atlas": my_cool_project
Please add a short description of the project: This analysis solves *all* the problems!
```

![`dso init "my_cool_project"`](docs/img/dso_init.png)

Within a project, you can use `dso create` to initalize folders and stages from a predefined template

```
$> dso create stage
? Choose a template: (Use arrow keys)
bash
» quarto
Please enter the name of the stage, e.g. "01_preprocessing": 02_quality_control
Please add a short description of the stage: Make a PCA to detect outliers
```
![`dso create stage --template bash --description 'Quality control'`](docs/img/dso_stage.png)

All of the above is also possible in an interactively guided text based way if you do not supply the parameters directly.

### How-to write and use config files
The config files in a project, subfolder or stage are the cornerstone of any reproducable analysis by minimising analysis configuration errors within related scripts. Additionally, config files reduce the time needed to modify your scripts when changing configurations such as p-value cutoffs, excluded samples, output directory, data input, and many more.
Expand All @@ -45,42 +37,31 @@ A config file of a project, subfolder, or stage contains all necessary parameter

In DSO two parameter files are given called `params.yaml` and `params.in.yaml`. `params.yaml` is an autogenerated YAML containing all the parameters specified in the params.in.yaml and other params.yaml files in its parent directories (see figure below for an example how this behaves in real). `params.yaml` will be compiled when running `dso compile-config`.

<img src="img/config.png" width="500" alt="Hierarchical configuration schema" />
<img src="docs/img/config.png" width="500" alt="Hierarchical configuration schema" />

```
$> dso compile-config
[08/22/24 20:53:43] INFO Detected /home/grst/my_cool_project as project root.
INFO Compiling a total of 2 config files.
INFO Configuration compiled successfully.
```
![`dso init "my_cool_project" --description "test" && cd my_cool_project && dso compile-config`](docs/img/dso_compile_config.png)

### Linting checks

Dso provides linting checks that detect common errors in analysis projects. Right now only few checks are implemented,
but more will be available in the future.

To run the linting checks manuall, execute
To run the linting checks manuall, execute `dso lint`:

```
$> dso lint
[08/22/24 20:53:43] INFO Compiled a list of 22 to be linted
```
![`dso init "my_cool_project" --description "test" && cd my_cool_project && dso lint`](docs/img/dso_lint.png)

However, it is preferable to execute linting checks as pre-commit hooks and/or as continuous integration checks.
A `.pre-commit-config.yaml` comes with the DSO project template. Simply activate it using `pre-commit install`.

### Reproducing projects

To reproduce/execute all stages within a project, run
To reproduce/execute all stages within a project, run `dso repro`

```
$> dso repro
```
![`dso init "my_cool_project" --description "test" && cd my_cool_project && dso repro`](docs/img/dso_repro.png)

This is a thin wrapper around `dvc repro` that compiles all configuration files beforehand.
DVC will only reproduce stages defined in the dvc.yaml where changes have been made. When dependencies have been changed, previous stages will also be re-run.


### Integration with quarto

DSO provides some additional tooling around quarto documents for generating reproducible reports. When you create a
Expand Down Expand Up @@ -112,7 +93,6 @@ To access stage parameters and resolve file paths relative to the stage director
companion package [`dso-r`](https://github.com/Boehringer-Ingelheim/dso-r) that provides the two functions
`read_params(stage_name)` and `stage_here(path)`.


## Installation

DSO requires Python 3.10 or later.
Expand Down
File renamed without changes
Binary file added docs/img/dso_compile_config.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/img/dso_init.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Binary file added docs/img/dso_lint.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/img/dso_repro.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/img/dso_stage.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.