Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug ci #131

Merged
merged 119 commits into from
Feb 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
d748a29
feat: Improving bash functionality, map catalog being specific to ite…
vijayvammi Jul 30, 2023
e759322
fix: release process PSR fix to 7.x.x
vijayvammi Jul 30, 2023
a67c8be
fix: release process PSR fix to 7.x.x
vijayvammi Jul 30, 2023
ee01e8c
fix: release process PSR fix to 7.x.x
vijayvammi Jul 30, 2023
2f8f262
feat: Working container task type
vijayvammi Jul 31, 2023
975c14d
feat: Adding required accomodations for local container
vijayvammi Aug 3, 2023
cac5010
feat: accommodating executor
vijayvammi Aug 3, 2023
a204729
feat: working container in local with data and parameters synced
vijayvammi Aug 5, 2023
d7b17f6
feat: Releasing container task type
vijayvammi Aug 5, 2023
ba67940
Merge branch 'main' into container
vijayvammi Aug 5, 2023
79e965f
feat: Improving SDK
vijayvammi Aug 7, 2023
ff71cd9
feat: Improving SDK
vijayvammi Aug 12, 2023
b5246c6
feat: Improving SDK
vijayvammi Aug 12, 2023
c2220f0
docs: Improving SDK
vijayvammi Aug 12, 2023
7ef392e
fix: WIP working with stricter mypy
vijayvammi Aug 16, 2023
ea3dd2c
fix: Stricter mypy WIP
vijayvammi Aug 16, 2023
b27616a
fix: Fixed mypy and test. To resume working on container related stuff
vijayvammi Aug 17, 2023
fdefc2d
feat: Final checks to container task type
vijayvammi Aug 20, 2023
c81058d
feat: Need to check job execution command in a few scenarios
vijayvammi Aug 20, 2023
530eaed
feat: Better .gitignore
vijayvammi Aug 25, 2023
2d80313
fix: WUP still in process
vijayvammi Aug 27, 2023
6325d1d
fix: merge conflict
vijayvammi Aug 28, 2023
ceb5b98
fix: mypy issues
vijayvammi Aug 28, 2023
a214012
fix: Adding service_type to BaseProviders
vijayvammi Aug 29, 2023
f192ea5
fix: updating sematic release
vijayvammi Aug 29, 2023
e6a9661
fix: todo for containers
vijayvammi Aug 29, 2023
ed8f441
fix: merging chat.tests
vijayvammi Aug 29, 2023
b6bd353
ci: figuring out the release
vijayvammi Aug 31, 2023
9d77f3b
ci: chaning magnus path
vijayvammi Aug 31, 2023
c5e3623
ci: fixing poetry install
vijayvammi Aug 31, 2023
47667d6
ci: forcing release
vijayvammi Aug 31, 2023
f3f0dfe
feat: Making integrations simple within magnus
vijayvammi Sep 1, 2023
1c1ce39
feat: Moving to Rich and adding examples
vijayvammi Sep 1, 2023
ef7a214
fix: Trying to get the logger working
vijayvammi Sep 1, 2023
7401019
fix: logging levels different for client code and magnus
vijayvammi Sep 1, 2023
1245a95
fix: Adding log level example
vijayvammi Sep 1, 2023
8241311
chore: Moving coverage to pyproject.toml, making integrations smoother
vijayvammi Sep 2, 2023
43abc4b
feat: Adding K8's extensions modules into magnus
vijayvammi Sep 2, 2023
f49a425
docs: Adding a bit more clear examples
vijayvammi Sep 3, 2023
5a66f1f
docs: Adding a bit more clear examples
vijayvammi Sep 3, 2023
7a2c9f9
feat: Getting pyinstaller into play
vijayvammi Sep 3, 2023
7dbfbd0
fix: remove docker build as it extends scope
vijayvammi Sep 5, 2023
a9d4a8e
fix: Moving to better statically typed code
vijayvammi Sep 6, 2023
b3a5180
fix: Refactor context
vijayvammi Sep 8, 2023
e9b54f6
feat: Proper base services and implemtations moving to extensions
vijayvammi Sep 10, 2023
e457903
feat: Moving to base vs implementation model
vijayvammi Sep 11, 2023
8e46f72
perf: Improving the start up time
vijayvammi Sep 12, 2023
82b1290
fix: Adding the test structure
vijayvammi Sep 12, 2023
e28078c
style: adding gitlint
vijayvammi Sep 12, 2023
8d94aea
fix: updating to pydantiv v2
vijayvammi Sep 14, 2023
3aa894a
fix: upgrade mypy and bring back standards
vijayvammi Sep 15, 2023
2fd9ca0
test: more and more tests
vijayvammi Sep 16, 2023
6a31eed
fix: Fixing pydantic nesting
vijayvammi Sep 18, 2023
b0e9d49
test: adding more tests
vijayvammi Sep 19, 2023
4d6d883
test: Still working on tests
vijayvammi Sep 19, 2023
bd5661d
test: working with tests and typing
vijayvammi Sep 21, 2023
0aa1907
test: added more tests, renamed as-is to stub
vijayvammi Sep 25, 2023
b51bef3
feat: in the process of adding a generic chunked data store
vijayvammi Sep 28, 2023
45f6443
fix: generic chunked data store
vijayvammi Sep 29, 2023
72ad870
feat: Bringing in argo
vijayvammi Oct 6, 2023
c1e7686
fix: W I P working on argo
vijayvammi Oct 9, 2023
9874527
fix: W I P working on argo
vijayvammi Oct 9, 2023
2605fed
feat: Brining in Argo
vijayvammi Oct 13, 2023
8388038
fix: mypy fixes
vijayvammi Oct 14, 2023
5bdfac0
feat: Adding SDK nodes and testing
vijayvammi Oct 25, 2023
12a5dd8
feat: Adding SDK nodes and testing
vijayvammi Oct 25, 2023
2721f6c
feat: workking on parallel sdk
vijayvammi Oct 28, 2023
d249244
feat: working on SDK, still wrok inprogress
vijayvammi Nov 6, 2023
88c9974
feat: Working SDK which looks good
vijayvammi Nov 6, 2023
d03a9a9
fix: TODO about the catalog
vijayvammi Nov 7, 2023
7286b2f
feat: improved testing and system to bring in other features
vijayvammi Nov 11, 2023
399b0d4
docs: Add docs
vijayvammi Nov 16, 2023
7f06161
docs: Adding more docs and making it better
vijayvammi Nov 17, 2023
1a03501
feat: Improved parameters, still working on it
vijayvammi Nov 19, 2023
19228be
feat: Improved parameter passing
vijayvammi Nov 24, 2023
b606065
docs: improved documentation
vijayvammi Nov 24, 2023
3565b1d
feat: Shell can now interact with paramters
vijayvammi Dec 14, 2023
ff60e7f
feat: improved secrets manager
vijayvammi Dec 15, 2023
e95f858
docs: more and more docs
vijayvammi Dec 21, 2023
2d1951a
feat: Improving docs and examples
vijayvammi Dec 29, 2023
ca4c5fb
feat: Improving docs and examples
vijayvammi Dec 30, 2023
a5c8d03
test: fixed broken tests
vijayvammi Jan 4, 2024
443c727
test: passing tests
vijayvammi Jan 4, 2024
e79c058
fix: Fixing catalog settings
vijayvammi Jan 4, 2024
6029841
test: Adding all examples as tests
vijayvammi Jan 4, 2024
793b052
feat: experiment tracking
vijayvammi Jan 9, 2024
d80b86c
feat: incrememntal experiment tracking
vijayvammi Jan 9, 2024
7235d7b
feat: incrememntal experiment tracking
vijayvammi Jan 9, 2024
858c4df
feat: incrememntal experiment tracking
vijayvammi Jan 9, 2024
0b62e4c
feat: mlflow integration
vijayvammi Jan 10, 2024
e95d446
feat: experiment tracking and more docs
vijayvammi Jan 10, 2024
53a5c9e
feat: working local container
vijayvammi Jan 15, 2024
c55af69
feat: still need to make tests pass
vijayvammi Jan 15, 2024
2998209
feat: adding map to sdk, still working
vijayvammi Jan 16, 2024
3d7c156
feat: map in sdk
vijayvammi Jan 16, 2024
30ca73b
feat: map in sdk
vijayvammi Jan 18, 2024
f0a2719
feat: map in sdk
vijayvammi Jan 18, 2024
4fb9c1f
feat: dynamic configuration file for sdk
vijayvammi Jan 18, 2024
99139c3
feat: Deeply nested SDK
vijayvammi Jan 18, 2024
e47564a
feat: brining in local container
vijayvammi Jan 22, 2024
ef14299
feat: brining in local container
vijayvammi Jan 22, 2024
8759a0e
feat: Brining in step overrides
vijayvammi Jan 23, 2024
00d0417
feat: Bringing in argo, still working
vijayvammi Jan 26, 2024
39cd987
feat: Bringing in argo
vijayvammi Jan 29, 2024
f94e49a
docs: more and more docs
vijayvammi Feb 5, 2024
d76cf86
feat: Adding mocked executor
vijayvammi Feb 10, 2024
d67e675
feat: mocked executors
vijayvammi Feb 12, 2024
ef5d963
docs: trying mermaid
vijayvammi Feb 12, 2024
cfca1bb
docs: trying mermaid
vijayvammi Feb 12, 2024
d7acda6
docs: trying mermaid
vijayvammi Feb 12, 2024
9fc0c27
docs: trying mermaid
vijayvammi Feb 12, 2024
ce7613d
docs: trying mermaid
vijayvammi Feb 12, 2024
8793f59
docs: trying mermaid
vijayvammi Feb 12, 2024
e00deba
docs: trying mermaid
vijayvammi Feb 12, 2024
5c155aa
docs: trying mermaid
vijayvammi Feb 12, 2024
bba76ad
feat: Getting a bit closer to release
vijayvammi Feb 14, 2024
557393e
docs: Updated references
vijayvammi Feb 14, 2024
0168dfe
fix: broken test
vijayvammi Feb 14, 2024
d70889a
ci: release is broken
vijayvammi Feb 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 0 additions & 7 deletions .coveragerc

This file was deleted.

16 changes: 16 additions & 0 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,22 @@ jobs:
with:
python-version: 3.8
- run: python -m pip install poetry
- run: |
# Download the binary
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/argo-linux-amd64.gz

# Unzip
gunzip argo-linux-amd64.gz

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo

# Test installation
argo version

- run: |
python -m poetry install
poetry run tox
38 changes: 35 additions & 3 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
- "magnus/**"
branches:
- "main"
- "rc"

jobs:
PRCheck:
Expand All @@ -13,10 +14,26 @@ jobs:
- uses: actions/setup-python@v4
with:
python-version: 3.8
- run: |
# Download the binary
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/argo-linux-amd64.gz

# Unzip
gunzip argo-linux-amd64.gz

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo

# Test installation
argo version
- run: python -m pip install poetry
- run: |
python -m poetry install
poetry run tox

Release:
runs-on: ubuntu-latest
needs: PRCheck
Expand All @@ -27,15 +44,28 @@ jobs:
- uses: actions/setup-python@v4
with:
python-version: 3.8
- run: python -m pip install python-semantic-release==7.34.6
- run: python -m pip install python-semantic-release==8.0.7
- name: Figure version
continue-on-error: true
id: last_tag
run: |

VERSION=$(semantic-release print-version --patch)
echo $VERSION
CURRENT=$(git tag --sort=-committerdate -l | head -n 1)
echo "current: $CURRENT"

VERSION=$(python -m semantic-release --noop --strict version --no-push --no-commit --print)
echo "New: $VERSION"

if [ "$CURRENT" == "$VERSION" ]; then
echo "version="" >> $GITHUB_OUTPUT
exit 1
fi

echo "version=$VERSION" >> $GITHUB_OUTPUT
exit 0

- name: Apply new tag
if: steps.last_tag.outcome == 'success'
env:
VERSION: ${{ steps.last_tag.outputs.version }}
uses: actions/github-script@v6
Expand All @@ -50,6 +80,7 @@ jobs:
sha: context.sha
})
- name: Publish to PyPI
if: steps.last_tag.outcome == 'success'
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
LAST_TAG: ${{ steps.last_tag.outputs.version }}
Expand All @@ -60,6 +91,7 @@ jobs:
poetry publish --build

- name: "Create release"
if: steps.last_tag.outcome == 'success'
env:
RELEASE_TAG: ${{ steps.last_tag.outputs.version }}
uses: "actions/github-script@v6"
Expand Down
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -141,3 +141,16 @@ cython_debug/

# VSCode specific settings
.vscode


.run_log_store/

.catalog/

cov.xml

.DS_Store

data/

example_bak/
22 changes: 20 additions & 2 deletions .mypy.ini
Original file line number Diff line number Diff line change
@@ -1,6 +1,24 @@
[mypy]
implicit_optional = True
ignore_missing_imports = True
exclude = _*FF.py
ignore_missing_imports = True
plugins = pydantic.mypy
show_error_codes = True

follow_imports = silent
warn_redundant_casts = True
warn_unused_ignores = True
#disallow_any_generics = True # This complicates it a lot
check_untyped_defs = True
implicit_reexport = True

# for strict mypy: (this is the tricky one :-))
#disallow_untyped_defs = True

[pydantic-mypy]
init_forbid_extra = True
init_typed = True
warn_required_dynamic_aliases = True
warn_untyped_fields = True

[mypy-ruamel.*]
ignore_missing_imports = True
Expand Down
13 changes: 13 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,16 @@ repos:
# pre-commit's default_language_version, see
# https://pre-commit.com/#top_level-default_language_version
language_version: python3.8

- repo: https://github.com/jorisroovers/gitlint
rev: v0.19.1
hooks:
- id: gitlint
args:
[
--contrib=CT1,
--ignore=title-trailing-punctuation,
--ignore=T3,
--ignore=B6,
--msg-filename,
]
Loading
Loading