Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic implementation #1

Merged
merged 69 commits into from
Mar 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
b0088db
Initial commit after migration to combined repo
strieflin Mar 11, 2022
71e1105
Launch generator pod
strieflin Mar 11, 2022
d23f3aa
Upload tuples and track task status
strieflin Apr 8, 2022
7fad059
Rename task states to avoid name clashes
strieflin Apr 8, 2022
73b2b96
Track job state based on task states
strieflin Apr 8, 2022
2f7a4c0
Move activation to job conroller
strieflin Apr 8, 2022
0fdda5a
Use compliant base images from carbynestack/base-images
strieflin May 6, 2022
d4e05b3
Fix build issues resulting from deprecation of insecure schemes by Gi…
strieflin May 6, 2022
8a0bc5d
Compute prng seed from md5sum of job id
strieflin May 6, 2022
64b7580
Add pre-commit configuration
strieflin May 6, 2022
cd20989
Add basic scheduler
strieflin May 13, 2022
b1f9eb2
Add SBOM creation script and 3rd party licenses
strieflin May 20, 2022
31c5cde
Use latest CLI version in provisioner
strieflin May 20, 2022
acb4593
Add readme for fake-offline crg
strieflin May 20, 2022
1b03e71
Move deploy scripts to hack folder
strieflin May 20, 2022
825aa4c
Add architecture and usage instructions to README
strieflin May 20, 2022
b4bfeef
Update copyright headers
strieflin May 20, 2022
e140000
Add copyright headers to sample manifests
strieflin May 20, 2022
e5594d9
Change type for count property of job
strieflin May 20, 2022
01e7d44
Add README to provisioner
strieflin May 20, 2022
1074963
Add / revise READMEs in submodules
strieflin May 20, 2022
fc98c5c
Use compliant base image in fake CRG, fix bugs in KII implementation
strieflin May 20, 2022
f577e69
Include AUTHORS files when collecting 3rd party license information
strieflin May 20, 2022
c270633
Add license header to .gitignore
strieflin May 20, 2022
49f813c
Add Codacy badge
strieflin May 20, 2022
ad228af
Fix first batch of SCA findings
strieflin May 23, 2022
543054a
Add EOL pre-commit hook
strieflin May 23, 2022
52ccfbf
Implement improvements from review
strieflin Jun 24, 2022
885c15e
Add tests
strieflin Jul 1, 2022
306af57
Improve tests and refactor codebase
strieflin Jul 29, 2022
0f7eb64
Verify eventual proxy task deletion
strieflin Jul 29, 2022
f9ae3aa
Update SBOM and fullfil OSS obligations
strieflin Aug 2, 2022
c02217f
Fix SCA findings
strieflin Aug 2, 2022
43e364b
Move castor client to separate package
strieflin Aug 2, 2022
a8a0f62
Add documentation to Castor client and improve logging / error handling
strieflin Aug 2, 2022
5129e59
Add documentation to etcd key implementation
strieflin Aug 2, 2022
967a1c3
Reenable castor client tests after moving to dedicated package
strieflin Aug 2, 2022
1284066
Improve controller test and TTL for jobs
strieflin Sep 9, 2022
1a32ccd
Add package documentation
strieflin Sep 9, 2022
cb859c4
Address minor review comments
strieflin Sep 9, 2022
ff24388
Improve hack script logic
strieflin Dec 16, 2022
ab32c58
Refactor flag handling and make etcd endpoint configurable
strieflin Dec 16, 2022
afde289
Improve documentation for configuration resources
strieflin Dec 16, 2022
866d9a3
Support for generic entrypoint instead of kii-run.sh script
strieflin Dec 16, 2022
c85c85e
Improve KII documentation by removing MP-SPDZ CRG specific information
strieflin Dec 16, 2022
f43f252
Make generator image configurable and replace CMD by ENTRYPOINT in pr…
strieflin Dec 16, 2022
19f22e0
Improve logging
strieflin Dec 16, 2022
32cee0b
Add target folders to .gitignore
strieflin Dec 16, 2022
d959197
Refator code to use strategy pattern for scheduling logic
strieflin Dec 20, 2022
d85d27d
Improve package documentation
strieflin Dec 20, 2022
d043c9f
Add initial GHA build logic
strieflin Dec 20, 2022
c42d6f6
Rename from klyshko-go to klyshko
strieflin Dec 20, 2022
2f38f7f
Ignore bundle-related files generated by Operator SDK
strieflin Dec 20, 2022
09be11b
Fix typo in copyright headers
strieflin Dec 20, 2022
b68ff6a
Add test step to operator build
strieflin Dec 20, 2022
526f40c
Add helm chart for klyshko-operator
strieflin Dec 21, 2022
9650963
Add missing timestamp in job status on creation
strieflin Dec 21, 2022
a7799ed
Fix non-working helm chart
strieflin Dec 21, 2022
fee0129
Make provisioner image configurable
strieflin Dec 21, 2022
8eee93e
Update README with deploy instructions for helm
strieflin Jan 27, 2023
73cb3a2
Push image pull secrets to top-level configuration
strieflin Jan 27, 2023
d26cf88
Update pre-commit configuration with new checks and bump versions
strieflin Jan 27, 2023
16c1487
Address codacy findinds
strieflin Jan 27, 2023
d13c2ae
Add codecov badge
strieflin Jan 27, 2023
7bd3d38
Address review comments
strieflin Mar 7, 2023
e63bcc8
chore: add commitlint pre-commit hook
strieflin Mar 10, 2023
7ea2274
chore: introduce release process
strieflin Mar 10, 2023
52fd89b
chore: add prettier hook to pre-commit
strieflin Mar 11, 2023
dcbe01f
chore(operator): add source link for MPL2.0-versioned dependency
strieflin Mar 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .github/workflows/mp-spdz.publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Publish MP-SPDZ
on:
push:
tags:
- "mp-spdz-v[0-9]+.[0-9]+.[0-9]+"
defaults:
run:
working-directory: klyshko-mp-spdz
env:
REGISTRY: ghcr.io
IMAGE_NAME: carbynestack/klyshko-mp-spdz
WORKING_DIRECTORY: klyshko-mp-spdz
jobs:
publish:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Retrieve license obligation resources
run: |
cd 3RD-PARTY-LICENSES
find . -maxdepth 1 -type d -not -path . | zip -r@ 3rd-party-copyrights
find . -iname origin.src | \
awk '{ \
split($0,b,"/"); \
system("xargs < " $0 " curl --create-dirs -Lo ./sources/" b[2] ".zip " $2)}' && \
find -regex './sources$' | awk '{system("zip -jr ./3rd-party-sources.zip " $0)}'
mkdir -p ../license-obligations && mv `find . -regex "^./3rd-party-.*.zip$"` ../license-obligations/
- name: Update Release with license obligations resources
uses: ncipollo/release-action@v1
with:
allowUpdates: true
artifacts: ${{ env.WORKING_DIRECTORY }}/license-obligations/*
artifactErrorsFailBuild: true
makeLatest: true
omitBodyDuringUpdate: true
omitNameDuringUpdate: true
- name: Log in to container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=match,pattern=mp-spdz-v(\d+.\d+.\d+),group=1
- name: Build and push Docker image
uses: docker/build-push-action@v3
with:
context: ${{ env.WORKING_DIRECTORY }}
file: ${{ env.WORKING_DIRECTORY }}/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
38 changes: 38 additions & 0 deletions .github/workflows/operator-chart.publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Publish Operator Chart
on:
push:
tags:
- "operator-chart-v[0-9]+.[0-9]+.[0-9]+"
env:
REGISTRY: ghcr.io
WORKING_DIRECTORY: klyshko-operator
jobs:
publish:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Get tag
run: echo "TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
- name: Log in to container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push Helm Chart
uses: appany/[email protected]
with:
name: klyshko-operator
repository: carbynestack
tag: ${{ env.TAG }}
path: ${{ env.WORKING_DIRECTORY }}/charts/klyshko-operator
registry: ${{ env.REGISTRY }}
registry_username: ${{ github.actor }}
registry_password: ${{ secrets.GITHUB_TOKEN }}
59 changes: 59 additions & 0 deletions .github/workflows/operator.build-and-test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Build and test Operator
on:
workflow_dispatch:
push:
branches:
- 'master'
pull_request:
branches:
- 'master'
jobs:
changes:
runs-on: ubuntu-22.04
permissions:
pull-requests: read
outputs:
operator: ${{ steps.filter.outputs.operator }}
steps:
- name: Check whether Operator codebase (w/o chart) is affected
uses: dorny/paths-filter@v2
id: filter
with:
filters: |
operator:
- 'klyshko-operator/!(charts/**)/**'
test:
runs-on: ubuntu-22.04
needs: changes
if: ${{ needs.changes.outputs.operator == 'true' }}
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.16'
- name: Setup Build Cache
uses: actions/cache@v3
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
- name: Build and Test
run: |
cd klyshko-operator
make test
- name: Publishing Coverage
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
name: codecov
64 changes: 64 additions & 0 deletions .github/workflows/operator.publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Publish Operator
on:
push:
tags:
- "operator-v[0-9]+.[0-9]+.[0-9]+"
defaults:
run:
working-directory: klyshko-operator
env:
REGISTRY: ghcr.io
IMAGE_NAME: carbynestack/klyshko-operator
WORKING_DIRECTORY: klyshko-operator
jobs:
publish:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Retrieve license obligation resources
run: |
cd 3RD-PARTY-LICENSES
find . -maxdepth 1 -type d -not -path . | zip -r@ 3rd-party-copyrights
find . -iname origin.src | \
awk '{ \
split($0,b,"/"); \
system("xargs < " $0 " curl --create-dirs -Lo ./sources/" b[2] ".zip " $2)}' && \
find -regex './sources$' | awk '{system("zip -jr ./3rd-party-sources.zip " $0)}'
mkdir -p ../license-obligations && mv `find . -regex "^./3rd-party-.*.zip$"` ../license-obligations/
- name: Update Release with license obligations resources
uses: ncipollo/release-action@v1
with:
allowUpdates: true
artifacts: ${{ env.WORKING_DIRECTORY }}/license-obligations/*
artifactErrorsFailBuild: true
makeLatest: true
omitBodyDuringUpdate: true
omitNameDuringUpdate: true
- name: Log in to container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=match,pattern=operator-v(\d+.\d+.\d+),group=1
- name: Build and push Docker image
uses: docker/build-push-action@v3
with:
context: ${{ env.WORKING_DIRECTORY }}
file: ${{ env.WORKING_DIRECTORY }}/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
64 changes: 64 additions & 0 deletions .github/workflows/provisioner.publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Publish Provisioner
on:
push:
tags:
- "provisioner-v[0-9]+.[0-9]+.[0-9]+"
defaults:
run:
working-directory: klyshko-provisioner
env:
REGISTRY: ghcr.io
IMAGE_NAME: carbynestack/klyshko-provisioner
WORKING_DIRECTORY: klyshko-provisioner
jobs:
publish:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Retrieve license obligation resources
run: |
cd 3RD-PARTY-LICENSES
find . -maxdepth 1 -type d -not -path . | zip -r@ 3rd-party-copyrights
find . -iname origin.src | \
awk '{ \
split($0,b,"/"); \
system("xargs < " $0 " curl --create-dirs -Lo ./sources/" b[2] ".zip " $2)}' && \
find -regex './sources$' | awk '{system("zip -jr ./3rd-party-sources.zip " $0)}'
mkdir -p ../license-obligations && mv `find . -regex "^./3rd-party-.*.zip$"` ../license-obligations/
- name: Update Release with license obligations resources
uses: ncipollo/release-action@v1
with:
allowUpdates: true
artifacts: ${{ env.WORKING_DIRECTORY }}/license-obligations/*
artifactErrorsFailBuild: true
makeLatest: true
omitBodyDuringUpdate: true
omitNameDuringUpdate: true
- name: Log in to container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=match,pattern=provisioner-v(\d+.\d+.\d+),group=1
- name: Build and push Docker image
uses: docker/build-push-action@v3
with:
context: ${{ env.WORKING_DIRECTORY }}
file: ${{ env.WORKING_DIRECTORY }}/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
20 changes: 20 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#
# Copyright (c) 2023 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#
name: Release
on:
push:
branches:
- master
jobs:
release-please:
runs-on: ubuntu-22.04
steps:
- name: Release Please
uses: google-github-actions/release-please-action@v3
with:
command: manifest
token: ${{ secrets.CS_MINION_PAT }}
15 changes: 14 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
#
# Copyright (c) 2022 - for information on the respective copyright owner
# see the NOTICE file and/or the repository https://github.com/carbynestack/klyshko.
#
# SPDX-License-Identifier: Apache-2.0
#

# IntelliJ
.idea
*.ipr
*.iml
*.iws
*.iws

# Go
/klyshko-operator/vendor/

# Java
**/target
2 changes: 1 addition & 1 deletion .markdownlint.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
"MD043": {
"headings": null
}
}
}
37 changes: 33 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,39 @@
# SPDX-License-Identifier: Apache-2.0
#
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
- id: check-byte-order-marker
- id: check-yaml
exclude: ^klyshko-operator/charts/klyshko-operator/templates/.*|klyshko-operator/config/manager/manager.yaml$
- id: end-of-file-fixer
- id: trailing-whitespace
- id: mixed-line-ending
- repo: https://github.com/alessandrojcm/commitlint-pre-commit-hook
rev: v9.4.0
hooks:
- id: commitlint
stages:
- commit-msg
additional_dependencies:
- '@commitlint/config-conventional'
- repo: https://github.com/tekwizely/pre-commit-golang
rev: v1.0.0-beta.5
rev: v1.0.0-rc.1
hooks:
- id: go-fmt
args: [-w] # Update source files directly
- repo: https://github.com/koalaman/shellcheck-precommit
rev: v0.7.2
hooks:
- id: shellcheck
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v2.7.1
hooks:
- id: prettier
types_or: [javascript]
- repo: https://github.com/executablebooks/mdformat
rev: 0.7.14
rev: 0.7.16
hooks:
- id: mdformat
args:
Expand All @@ -21,7 +47,10 @@ repos:
- mdformat-gfm
exclude: ^3RD-PARTY-LICENSES/.*$
- repo: https://github.com/igorshubovych/markdownlint-cli
rev: v0.31.1
rev: v0.33.0
hooks:
- id: markdownlint
exclude: ^3RD-PARTY-LICENSES/.*$
args:
- --disable
- MD013
exclude: ^3RD-PARTY-LICENSES/.*$
Loading