Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental CI changes #294

Merged
merged 27 commits into from
Nov 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 87 additions & 138 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,164 +1,113 @@
version: 2.1

parameters:
preview:
type: boolean
default: true
3-10:
type: boolean
default: true
3-11:
type: boolean
default: true
# orbs:
# coveralls: coveralls/[email protected]

workflows:
ci:
jobs:
- lint
- test:
matrix:
parameters:
# TODO: Revisit why pyenv doesn't recognize 3.12
python_version: ["3.8", "3.9", "3.10", "3.11"] # "3.12"
Comment on lines +13 to +14
Copy link
Member Author

@aMahanna aMahanna Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason pyenv 2.3.37 does not recognize 3.12 when attempting to do pyenv install 3.12...

However, pyenv install 3.12 worked fine locally with pyenv 2.3.31 🤔

will revisit

arangodb_config: ["single", "cluster"]
arangodb_license: ["community", "enterprise"]
Copy link
Member Author

@aMahanna aMahanna Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want to clarify - according to the current state of starter.sh:

community --> docker/arangodb (https://hub.docker.com/r/arangodb/arangodb/tags)
enterprise --> docker/enterprise (https://hub.docker.com/r/arangodb/enterprise/tags)

Perhaps we can revisit the idea of community-preview (https://hub.docker.com/r/arangodb/arangodb-preview/tags) and enterprise-preview (https://hub.docker.com/r/arangodb/enterprise-preview/tags)

arangodb_version: ["3.10.10", "3.11.4", "latest"]

jobs:
run-tests:
resource_class: small
lint:
docker:
- image: python:latest
steps:
- checkout
- run:
name: Install Dependencies
command: pip install .[dev]

- run:
name: Run black
command: black --check --verbose --diff --color --config=pyproject.toml ./arango ./tests/

- run:
name: Run flake8
command: flake8 ./arango ./tests

- run:
name: Run isort
command: isort --check ./arango ./tests

- run:
name: Run mypy
command: mypy ./arango

test:
parameters:
python-version:
python_version:
type: string
default: "latest"
arangodb-version:
arangodb_config:
type: string
default: "arangodb:latest"
arangodb-config:
arangodb_license:
type: string
default: "single.conf"
cluster:
type: boolean
default: false
enterprise:
type: boolean
default: false
docker:
- image: python:<< parameters.python-version >>
command: ["/bin/sh", "-c", "python -m http.server"]
- image: arangodb/<< parameters.arangodb-version >>
environment:
ARANGODB_CONF: << parameters.arangodb-config >>
PROJECT: /root/project
command:
- "/bin/sh"
- "-c"
- >
while ! wget -q -O /dev/null http://localhost:8000/$PROJECT/tests/static/setup.sh; do sleep 1; done &&
wget -O - http://localhost:8000/$PROJECT/tests/static/setup.sh |
/bin/sh
arangodb_version:
type: string
# TODO: Reconsider using a docker image instead of a machine
# i.e cimg/python:<< parameters.python_version >>
machine:
image: ubuntu-2204:current
Comment on lines +55 to +58
Copy link
Member Author

@aMahanna aMahanna Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully we can revisit using cimg/python with starter.sh as that would cut the job times from ~4 minutes to ~2 minutes.

We currently rely on ubuntu-2204 in combination with pyenv, which is a ~2min bottleneck:

https://github.com/ArangoDB-Community/python-arango/blob/eb8ce582e7b36acf71a51f1de43220c84842e133/.circleci/config.yml#L71-L77

steps:
- checkout

- run:
name: "Install Dependencies"
name: Set Up ArangoDB
command: |
pip install -e .[dev]
chmod +x starter.sh
./starter.sh << parameters.arangodb_config >> << parameters.arangodb_license >> << parameters.arangodb_version >>

- restore_cache:
key: pip-and-local-cache

# TODO: Revisit this bottleneck
- run:
name: "Wait for ArangoDB starter"
name: Setup Python
command: |
wget --quiet --waitretry=1 --tries=120 -O - http://localhost:8528/version
if [ $? -eq 0 ]; then
echo "starter ready"
exit 0
else
echo "starter not ready, giving up"
exit 1
fi
pyenv --version
pyenv install -f << parameters.python_version >>
pyenv global << parameters.python_version >>

- run:
name: "Install Dependencies"
command: pip install -e .[dev]

- run: docker ps -a

- run:
name: "Run pytest"
command: |
mkdir test-results
args=("--junitxml=test-results/junit.xml" "--log-cli-level=DEBUG" "--host" "localhost")
if [ << parameters.cluster >> = true ]; then
args+=("--cluster" "--port=8529" "--port=8539" "--port=8549")
else
args+=("--port=8529")

args=("--junitxml=test-results/junit.xml" "--log-cli-level=DEBUG" "--host" "localhost" "--port=8529")
if [ << parameters.arangodb_config >> = "cluster" ]; then
args+=("--cluster" "--port=8539" "--port=8549")
fi
if [ << parameters.enterprise >> = true ]; then

if [ << parameters.arangodb_license >> = "enterprise" ]; then
args+=("--enterprise")
fi
echo "Running py.test with args: ${args[@]}"
py.test "${args[@]}"

echo "Running pytest with args: ${args[@]}"
pytest --cov=arango --cov-report=xml "${args[@]}"

- store_artifacts:
path: test-results

- store_test_results:
path: test-results

workflows:
python-3.8-community-single-3.10:
when: << pipeline.parameters.3-10 >>
jobs:
- run-tests:
name: python-3.8-community-single-3.10
python-version: "3.8.2"
arangodb-version: "arangodb:3.10.10"
arangodb-config: "single-3.10.conf"
cluster: false
enterprise: false
python-3.8-enterprise-cluster-3.10:
when: << pipeline.parameters.3-10 >>
jobs:
- run-tests:
name: python-3.8-enterprise-cluster-3.10
python-version: "3.8.2"
arangodb-version: "enterprise:3.10.10"
arangodb-config: "cluster-3.10.conf"
cluster: true
enterprise: true
python-3.10-community-single-3.11:
when: << pipeline.parameters.3-11 >>
jobs:
- run-tests:
name: python-3.10-community-single-3.11
python-version: "3.10.6"
arangodb-version: "arangodb:3.11.4"
arangodb-config: "single.conf"
cluster: false
enterprise: false
python-3.10-community-cluster-3.11:
when: << pipeline.parameters.3-11 >>
jobs:
- run-tests:
name: python-3.10-community-cluster-3.11
python-version: "3.10.6"
arangodb-version: "arangodb:3.11.4"
arangodb-config: "cluster.conf"
cluster: true
enterprise: false
python-3.10-enterprise-single-3.11:
when: << pipeline.parameters.3-11 >>
jobs:
- run-tests:
name: python-3.10-enterprise-single-3.11
python-version: "3.10.6"
arangodb-version: "enterprise:3.11.4"
arangodb-config: "single.conf"
cluster: false
enterprise: true
python-3.10-enterprise-cluster-3.11:
when: << pipeline.parameters.3-11 >>
jobs:
- run-tests:
name: python-3.10-enterprise-cluster-3.11
python-version: "3.10.6"
arangodb-version: "enterprise:3.11.4"
arangodb-config: "cluster.conf"
cluster: true
enterprise: true
python-latest-enterprise-single-preview:
when: << pipeline.parameters.preview >>
jobs:
- run-tests:
name: python-latest-enterprise-single-preview
python-version: "latest"
arangodb-version: "enterprise-preview:latest"
arangodb-config: "single.conf"
cluster: false
enterprise: true
python-latest-enterprise-cluster-preview:
when: << pipeline.parameters.preview >>
jobs:
- run-tests:
name: python-latest-enterprise-cluster-preview
python-version: "latest"
arangodb-version: "enterprise-preview:latest"
arangodb-config: "cluster.conf"
cluster: true
enterprise: true
# - run:
# name: Upload to Coveralls
# command: |
# if [ "<< parameters.python_version >>" = "3.11" && "<< parameters.arangodb_config >>" = "single" && "<< parameters.arangodb_license >>" = "community" && "<< parameters.arangodb_version >>" = "latest" ]; then
# coveralls/upload
# fi
Comment on lines +108 to +113
Copy link
Member Author

@aMahanna aMahanna Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uploading coverage to coveralls won't be possible until we request access from the Organization Admin:

image

Will figure out who that is..

52 changes: 0 additions & 52 deletions .github/workflows/build.yaml

This file was deleted.

Loading