diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 26d6723e0..d4d425695 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,18 +1,3 @@ - - - ## Pull Request Overview - Specify a list of summary items that this PR is accomplishing diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 341405d0c..974a118e2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,33 +2,16 @@ All contributions are welcome and encouraged. There are a few guidelines and styling aspects that we require and encourage you to use so that we might see this project through many years of successful development. -## Development Guidelines +## Pull Request Considerations -### Pull Request Checklist +Details on how to configure and pass each of these required checks is detailed in the sections in this guideline section. -To submit a pull request (PR) to Houston, we require the following standards to be enforced. Details on how to configure and pass each of these required checks is detailed in the sections in this guideline section. +### Pre-commit +We require code formatting with Brunette (a *better* version of Black) and linted with Flake8 using the pre-commit (includes automatic checks with brunette and flake8 plus includes other general line-ending, single-quote strings, permissions, and security checks). Pre-commit is installed once and then runs automatically. -* **Ensure that the PR is properly formatted** - * We require code formatting with Brunette (a *better* version of Black) and linted with Flake8 using the pre-commit (includes automatic checks with brunette and flake8 plus includes other general line-ending, single-quote strings, permissions, and security checks) - ``` - # Command Line Example - pre-commit run --all-files - ``` -* **Ensure that the PR is properly rebased** - * We require new feature code to be rebased onto the latest version of the `develop` branch. - ``` - git fetch -p - git checkout - git diff origin/.. # Check there's no difference between local branch and remote branch - git rebase -i origin/main # "Pick" all commits in feature branch - # Resolve all conflicts - git log --graph --oneline --decorate origin/main HEAD # feature-branch commits should be on top of origin/develop - git show --reverse origin/main.. # Check the changes in each commit if necessary - git push --force origin - ``` -* **Ensure that the PR uses a consolidated database migration** - * We require new any database migrations (optional) with Alembic are consolidated and condensed into a single file and version. Exceptions to this rule (allowing possibly up to 3) will be allowed after extensive discussion and justification. - * All database migrations should be created using a downgrade revision that matches the existing revision used on the `develop` branch. Further,a PR should never be merged into develop that contains multiple revision heads. +### Consolidated database migration +Database migrations (optional) with Alembic are consolidated and condensed into a single file and version. Exceptions to this rule (allowing possibly up to 3) will be allowed after extensive discussion and justification. +All database migrations should be created using a downgrade revision that matches the existing revision used on the `develop` branch. Further,a PR should never be merged into develop that contains multiple revision heads. ``` invoke app.db.downgrade rm -rf migrations/versions/.py @@ -37,21 +20,24 @@ To submit a pull request (PR) to Houston, we require the following standards to invoke app.db.history # Check that the history matches invoke app.db.heads # Ensure that there is only one head ``` -* **Ensure that the PR is properly tested** - * We require new feature code to be tested via Python tests and simulated REST API tests. We use PyTest to ensure that your code is working cohesively and that any new functionality is exercised. - * We require new feature code to also be fully compatible with a containerized runtime environment like Docker. + +### Testing +Feature code is tested via Python tests and simulated REST API tests. We use PyTest to ensure that your code is working cohesively and that any new functionality is exercised. +Feature code to also be fully compatible with a containerized runtime environment like Docker. ``` pytest ``` -* **Ensure that the PR is properly covered** - * We require new feature code to be tested (previous item) and that the percentage of the code covered by tests does not decrease. We use PyTest Coverage and CodeCov.io to ensure that your code is being properly covered by new tests. - * To export a HTML report of the current coverage statistics, run the following: + +### Test coverage +New feature code should have automated tests, and the percentage of the code covered by tests does not decrease with the addition of new features. We use PyTest Coverage and CodeCov.io to ensure that your code is being properly covered by new tests. +To export a HTML report of the current coverage statistics, run the following: ``` pytest -s -v --cov=./ --cov-report=html open _coverage/html/index.html ``` -* **Ensure that the PR is properly sanitized** - * We require the PR to not include large files (images, database files, etc) without using [GitHub Large File Store (LFS)](https://git-lfs.github.com). + +### Large file sanitation +PRs should not include large files (images, database files, etc) without using [GitHub Large File Store (LFS)](https://git-lfs.github.com). ``` git lfs install git lfs track "*.png" @@ -60,11 +46,7 @@ To submit a pull request (PR) to Houston, we require the following standards to git commit -m "Add new image file" git push ``` - * We also require any sensitive code, configurations, or pre-specified values be omitted, truncated, or redacted. For example, the file `_db/secrets/py` is not committed into the repository and is ignored by `.gitignore`. -* **Ensure that the PR is properly reviewed** - * After the preceding checks are satisfied, the code is ready for review. All PRs are required to be reviewed and approved by at least one registered contributor or administrator on the Houston project. - * When the PR is created in GitHub, make sure that the repository is specified as `WildMeOrg/houston` and not its original fork. Further, make sure that the base branch is specified as `develop` and not `master`. - +Any sensitive code, configurations, or pre-specified values should be omitted, truncated, or redacted. For example, the file `_db/secrets/py` is not committed into the repository and is ignored by `.gitignore`. ## Code Style @@ -91,10 +73,55 @@ Reference [pre-commit's installation instructions](https://pre-commit.com/#insta See `.pre-commit-config.yaml` for a list of configured linters and fixers. ## Development Environment +### Install from source for troubleshooting + +Installation of Houston and the other components of Codex from source is intended to facilitate development leveraging the docker-compose environment. Full deployment of Codex outside docker-compose orchestration is not supported, and any changes should not be considered finished until they have been tested in the docker-compose environment. + +#### Clone the Project + +```bash +git clone --recurse-submodules https://github.com/USERNAME/houston.git +cd houston/ +``` +#### Setup Codex Environment + +It is recommended to use virtualenv or Anaconda/Miniconda to manage Python +dependencies. Please, learn details yourself. +For quickstart purposes the following will set up a virtualenv for you: + +```bash +./scripts/codex/venv.sh +source virtualenv/houston3.7/bin/activate + +# To add bash-completion +export SCRIPT="$(pwd)/.invoke-completion.sh" +invoke --print-completion-script bash > $SCRIPT +echo "source $SCRIPT" >> virtualenv/houston3.7/bin/activate +``` + +Set up and install the package: + +```bash +invoke dependencies.install +``` + +#### Run Server + +NOTE: All dependencies and database migrations will be automatically handled, +so go ahead and turn the server ON! (Read more details on this in Tips section) + +```bash +export HOUSTON_APP_CONTEXT=codex +$ invoke app.run +``` + +#### Deploy Server + +In general, you deploy this app as any other Flask/WSGI application. There are +a few basic deployment strategies documented in the [`./deploy/`](./deploy/) +folder. -It's recommended that you install the Houston application and its dependencies [using the Docker container environment](#container-installation). If you'd like to make things difficult you can also look at the [local installation instructions](#local-installation). - ### Installing and running on your local system #### PyInvoke installation @@ -143,27 +170,55 @@ invoke dependencies.install invoke codex.run ``` +#### Usage +### Usage Tips - -### Installing and running with Docker +Once you have invoke, you can learn all available commands related to this +project from: -These instructions will install and run the application and dependent application. +```bash +$ invoke --list +``` -The configuration is by environment variable via the `.env` file in this directory. +Learn more about each command with the following syntax: -For development purposes, this setup exposes each of the services as follows: +```bash +$ invoke --help +``` - -- EDM - http://localhost:81/ -- Sage (Wildbook-IA) - http://localhost:82/ -- Houston - http://localhost:83/houston/ -- CODEX (frontend) - http://localhost:84/ -- CODEX (api docs) - http://localhost:84/api/v1/ -- GitLab - http://localhost:85 +For example: -See also the results of `docker-compose ps -a`, which will list the services and the ports exposed on those services. +```bash +$ invoke --help codex.run +Usage: inv[oke] [--core-opts] codex.run [--options] [other tasks here ...] + +Docstring: + Run DDOTS RESTful API Server. + +Options: + -d, --[no-]development + -h STRING, --host=STRING + -i, --[no-]install-dependencies + -p, --port + -u, --[no-]upgrade-db +``` -#### Usage +Use the following command to enter ipython shell (`ipython` must be installed): + +```bash +$ invoke app.env.enter +``` + +`codex.run` and `app.env.enter` tasks automatically prepare all dependencies +(using `pip install`) and migrate database schema to the latest version. + +Database schema migration is handled via `app.db.*` tasks group. The most +common migration commands are `app.db.upgrade` (it is automatically run on +`codex.run`), and `app.db.migrate` (creates a new migration). + +You can use [`better_exceptions`](https://github.com/Qix-/better-exceptions) +package to enable detailed tracebacks. Just add `better_exceptions` to the +`app/requirements.txt` and `import better_exceptions` in the `app/__init__.py`. ##### Running the applications @@ -290,7 +345,7 @@ in the `docker-compose.yml` by altering the `dev-frontend` volume mapping `- _fr More details about Codex front end contribution are outside the scope of this README but can be found here: [**codex-frontend**](https://github.com/WildMeOrg/codex-frontend) ##### EDM -The EDM is a compiled Java application, and no volume mapping solution to a running Docker container is available at this time. +The EDM is presently maintained for migration of production platforms from Wildbook to Codex. It will be removed when production platforms have all been migrated. ### Testing diff --git a/README.md b/README.md index ca8262e7a..33c4640c6 100644 --- a/README.md +++ b/README.md @@ -11,22 +11,95 @@ For a high-level explanation of the application in relation to other CODEX appli ## About this implementation -This project showcases my vision on how the RESTful API server should be -implemented. - -The goals that were achieved in this example: - * RESTful API server should be self-documented using OpenAPI (fka Swagger) specifications, so interactive documentation UI is in place * Authentication is handled with OAuth2 and using Resource Owner Password Credentials Grant (Password Flow) for first-party clients makes it usable not only for third-party "external" apps * Permissions are handled (and automaticaly documented) * PATCH method can be handled accordingly to [RFC 6902](http://tools.ietf.org/html/rfc6902) * Extensive testing with good code coverage. - -The package Flask-RESTX has been patched (see `flask_restx_patched` folder), so it can handle Marshmallow schemas and Webargs arguments. - -## Code Style and Development Guidelines - -See [Contributing to Houston](CONTRIBUTING.md) +* The package Flask-RESTX has been patched (see `flask_restx_patched` folder), so it can handle Marshmallow schemas and Webargs arguments. + +## Pull Request Workflow + +### Fork Houston +To start, you will need to be signed in to your GitHub account, have admin access to your OS's terminal, and have Git installed. +1. From your browser, in the top right corner of the [Houston repo](https://github.com/WildMeOrg/houston), click the **Fork** button. Confirm to be redirected to your own fork (check the url for your USERNAME in the namespace). +1. In your terminal, enter the command `git clone --recurse-submodules https://github.com/USERNAME/houston` +1. Once the houston directory become available in your working directory, move to it with the command `cd houston` +1. Add a reference to the original repo, denoting it as the upstream repo. +`git remote add upstream https://github.com/WildMeOrg/houston` +`git fetch upstream` + + +### Create Local Branch +You will want to work in a branch when doing any feature development you want to provide to the original project. +1. Verify you are on the main branch. The branch you have checked out will be used as the base for your new branch, so you typically want to start from main. +`git checkout main` +1. Create your feature branch. It can be helpful to include the issue number (ISSUENUMBER) you are working to address. +`git branch ISSUENUMBER-FEATUREBRANCHNAME` +1. Change to your feature branch so your changes are grouped together. +`git checkout ISSUENUMBER-FEATUREBRANCHNAME` +1. Update your branch (this is not needed if you just created new branch, but is a good habit to get into). +` git pull upstream main` + +### Making Local Changes +Make the code changes necessary for the issue you're working on. The following git commands may prove useful. + +* `git log`: lastest commits of current branch +* `git status`: current staged and unstaged modifications +* `git diff --staged`: the differences between the staging area and the last commit +* `git add : add files that have changes to staging in preparation for commit +* `git commit`: commits the stagged files, opens a text editor for you to write a commit log + +#### Using Docker for your Dev Environment +Set up the tools needed: +1. Ensure that you are in the `houston` directory +1. If you have not already, install docker ex: `sudo apt install docker` +1. If you have not already, install docker-compose ex: `sudo apt install docker-compose` +1. Install pre-commit to set up your linter. This will run automatically for any PR. +`pip install pre-commit` +1. Enter command `./scripts/codex/activate.sh` +1. `sudo sysctl -w vm.max_map_count=262144` + +Manage your container: +1. Enter `docker-compose up` to bring up the container. +1. In your browser, visit any of the following ports to confirm your system is running. + * Sage (Wildbook-IA) - http://localhost:82/ + * Houston - http://localhost:83/houston/ + * CODEX (frontend) - http://localhost:84/ + * CODEX (api docs) - http://localhost:84/api/v1/ +1. Enter `docker-compose down` to bring down the container. +1. To rebuild your docker image, enter `docker-compose up -build` + +#### App Setup +1. At http://localhost:84, work through the admin initial setup. +1. Navigate to Site Settings > Custom Fields +1. Add Species +1. Add Regions + +### Submit PR +Up to this point, all changes have been done to your local copy of Houston. You need to push the new commits to a remote branch to start the PR process. + +1. Now's the time clean up your PR if you choose to squash commits or rebase. If you're looking for more information on these practices, see this [pull request tutorial](https://yangsu.github.io/pull-request-tutorial/). +1. Push to the remote version of your branch ` git push ` +`git push origin ISSUENUMBER-FEATUREBRANCHNAME` +1. When prompted, provide your username and GitHub Personal Access Token. If you do not have a GitHub Personal Access Token, or do not have one with the correct permissions for your newly forked repository, you will need to [create a Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token). +1. Check the fork's page on GitHub to verify that you see a new branch with your added commits. You should see a line saying "This branch is X commits ahead" and a **Pull request** link. +1. Click the **Pull request** link to open a form that says "Able to merge". (If it says there are merge conflicts, go the [Wild Me Development Discord](https://discord.gg/zw4tr3RE4R) for help). +1. Use an explicit title for the PR and provide details in the comment area. Details can include text, or images, and should provide details as to what was done and why design decisions were made. +1. Click **Create a pull request**. + +### Respond to feedback +At this point, it's on us to get you feedback on your submission! Someone from the Wild Me team will review the project and provide any feedback that may be necessary. If changes are recommended, you'll need to checkout the branch you were working from, update the branch, and make these changes locally. + +1. `git checkout ISSUENUMBER-FEATUREBRANCHNAME` +1. `git pull upstream main` +1. Make required changes +1. `git add ` for all files impacted by changes +1. Determine which method would be most appropriate for updating your PR + * `git commit --ammend` if the changes are small stylistic changes + * `git commit` if the changes involved significant rework and require additional details + +See [Contributing to Houston](CONTRIBUTING.md) for code styles and other information. ## Project's Code Structure @@ -36,83 +109,10 @@ See [Project Structure](docs/project_file_structure.md) See [Background and Periodic Tasks](docs/background_tasks.md) -## Installation - -### Using docker-compose (recommended) - -#### Setup - -```bash -git clone --recurse-submodules https://github.com/WildMeOrg/houston.git - -# Option 1 - Activate Codex App -./scripts/codex/activate.sh -docker-compose up - -# Option 2 - Use Codex Config Explicitly -docker-compose -f docker-compose.codex.yml --env-file .env.codex up -``` - -Surf to http://localhost:84/. If you are having issues, see the [docker-compose debugging](docs/docker_compose_debugging.md) docs. - -### Installing from source - -#### Development Setup Note - -Installation of Houston and the other components of Codex from source is intended to facilitate development leveraging the docker-compose environment. - -See **Development Environment** section in [Contributing to Houston](CONTRIBUTING.md) -for details. Full deployment of Codex outside docker-compose orchestration is not supported, and any -changes should not be considered finished until they have been tested in the docker-compose environment. - -#### Clone the Project - -```bash -git clone --recurse-submodules https://github.com/WildMeOrg/houston.git -cd houston/ -``` - -#### Setup Codex Environment - -It is recommended to use virtualenv or Anaconda/Miniconda to manage Python -dependencies. Please, learn details yourself. -For quickstart purposes the following will set up a virtualenv for you: - -```bash -./scripts/codex/venv.sh -source virtualenv/houston3.7/bin/activate - -# To add bash-completion -export SCRIPT="$(pwd)/.invoke-completion.sh" -invoke --print-completion-script bash > $SCRIPT -echo "source $SCRIPT" >> virtualenv/houston3.7/bin/activate -``` - -Set up and install the package: - -```bash -invoke dependencies.install -``` - -#### Run Server - -NOTE: All dependencies and database migrations will be automatically handled, -so go ahead and turn the server ON! (Read more details on this in Tips section) - -```bash -export HOUSTON_APP_CONTEXT=codex -$ invoke app.run -``` - -#### Deploy Server - -In general, you deploy this app as any other Flask/WSGI application. There are -a few basic deployment strategies documented in the [`./deploy/`](./deploy/) -folder. +## Documentation -## Usage - +### Interactive Documentation Open online interactive API documentation: [http://127.0.0.1:5000/api/v1/](http://127.0.0.1:5000/api/v1/) @@ -121,55 +121,17 @@ Autogenerated swagger config is always available from NOTE: Use On/Off switch in documentation to sign in. -### Usage Tips - -Once you have invoke, you can learn all available commands related to this -project from: - -```bash -$ invoke --list -``` - -Learn more about each command with the following syntax: - -```bash -$ invoke --help -``` - -For example: +### Static documentation -```bash -$ invoke --help codex.run -Usage: inv[oke] [--core-opts] codex.run [--options] [other tasks here ...] - -Docstring: - Run DDOTS RESTful API Server. +To build and view the documentation use the following commands: -Options: - -d, --[no-]development - -h STRING, --host=STRING - -i, --[no-]install-dependencies - -p, --port - -u, --[no-]upgrade-db ``` - -Use the following command to enter ipython shell (`ipython` must be installed): - -```bash -$ invoke app.env.enter +cd docs +pip install -r requirements.txt +make html +open _build/html/index.html ``` -`codex.run` and `app.env.enter` tasks automatically prepare all dependencies -(using `pip install`) and migrate database schema to the latest version. - -Database schema migration is handled via `app.db.*` tasks group. The most -common migration commands are `app.db.upgrade` (it is automatically run on -`codex.run`), and `app.db.migrate` (creates a new migration). - -You can use [`better_exceptions`](https://github.com/Qix-/better-exceptions) -package to enable detailed tracebacks. Just add `better_exceptions` to the -`app/requirements.txt` and `import better_exceptions` in the `app/__init__.py`. - ## Dependencies ### Project Dependencies @@ -183,20 +145,6 @@ The [_tus_](https://tus.io) portions of this application require [**Redis**](htt [**Postgres**](https://www.postgresql.org/) is an optional dependency that can be used for a highly reliable scaled database solution. -## Documentation - -### Build the documentation - -To build and view the documentation use the following commands: - -``` -cd docs -pip install -r requirements.txt -make html -open _build/html/index.html -``` - - ## Site Settings There are some site settings that some features are dependent on: @@ -240,4 +188,4 @@ Settings can also be set via `SiteSetting`, the keys are ## License -This software is subject to the provisions of Apache License Version 2.0 (APL). See `LICENSE` for details. Copyright (c) 2020 Wild Me +This software is subject to the provisions of MIT License. See `LICENSE` for details. Copyright (c) 2023 Wild Me \ No newline at end of file diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 034e84803..000000000 --- a/SECURITY.md +++ /dev/null @@ -1,21 +0,0 @@ -# Security Policy - -## Supported Versions - -Use this section to tell people about which versions of your project are -currently being supported with security updates. - -| Version | Supported | -| ------- | ------------------ | -| 5.1.x | :white_check_mark: | -| 5.0.x | :x: | -| 4.0.x | :white_check_mark: | -| < 4.0 | :x: | - -## Reporting a Vulnerability - -Use this section to tell people how to report a vulnerability. - -Tell them where to go, how often they can expect to get an update on a -reported vulnerability, what to expect if the vulnerability is accepted or -declined, etc.