Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace test query results #302

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
7def8a5
Refactor code using handle_exceptions decorator
maxachis Apr 27, 2024
5698ab7
Black reformatting
maxachis Apr 27, 2024
3b2c241
update readme
josh-chamberlain Apr 30, 2024
68199c9
Add middleware tests for database interactions
maxachis May 3, 2024
3092634
Add testing helpers for database interactions
maxachis May 3, 2024
85ec2d7
Refactor test methods and introduce database testing helpers
maxachis May 3, 2024
94bf05a
Create (mostly) blank python_tests.yml file
maxachis May 3, 2024
8c7392d
Refactor test file and update test functions
maxachis May 6, 2024
0b86c49
Add middleware testing fixtures
maxachis May 6, 2024
ee11b16
Add tests for data source queries
maxachis May 6, 2024
16ebc65
Refactor pytest fixtures
maxachis May 6, 2024
197d80d
Added unit tests for archives_queries middleware
maxachis May 6, 2024
4a0efc8
Added unit tests for login_queries middleware
maxachis May 6, 2024
171b35b
Refactor and enhance get_most_recent_quick_search_query_log function
maxachis May 6, 2024
9a3ecf5
Add unit tests for quick search query middleware
maxachis May 6, 2024
23bee8b
Add unit tests for user and reset token query methods
maxachis May 6, 2024
e1fff5a
Remove redundant tests for middleware functions
maxachis May 6, 2024
6aa2170
Remove now-redundant tests
maxachis May 6, 2024
41deff0
Update README with revised testing instructions
maxachis May 6, 2024
e3e1e25
Rename Python test workflow and add configuration
maxachis May 6, 2024
b66aab9
Update test data source imports in middleware tests
maxachis May 6, 2024
f057548
Refactor test files and clean up imports
maxachis May 6, 2024
9767d9e
Improve code readability and ignore max line exceed warning
maxachis May 8, 2024
66e13d3
Update flake8 args and refine code formatting
maxachis May 8, 2024
a83bfc3
Update flake8 args and refine code formatting
maxachis May 8, 2024
6820412
Added custom exception for psycopg2 connection initialization
maxachis May 11, 2024
12d00c5
Handle UserNotFoundError in middleware and implement custom exception
maxachis May 11, 2024
c126810
Refactor query methods for data source handling
maxachis May 12, 2024
38070f0
Remove archives_put_query function and refactor related code
maxachis May 12, 2024
078d045
Update SQL commands and refactor unit tests
maxachis May 14, 2024
d752468
Refactor using black
maxachis May 14, 2024
f7a77f1
Update pytest command in pull.yaml workflow
maxachis May 14, 2024
9bb0d0f
Refactor using black
maxachis May 14, 2024
10dc54e
Add test skeletons for UserNotLoggedIn exception handling
maxachis May 14, 2024
a5c3747
Merge branch 'mc_testing_to_dev' into mc_issue_276_refactor_middlewar…
maxachis May 15, 2024
caaf833
Implement test for UserNotFoundError in login queries
maxachis May 15, 2024
bbb6edc
Merge branch 'mc_testing_to_dev' into mc_issue_181_middleware_god_fun…
maxachis May 15, 2024
9f99705
Refactor data access in tests and remove unused test
maxachis May 15, 2024
bc4f9bc
Add type hints and docstring to 'convert_data_source_matches' function
maxachis May 15, 2024
2132d0a
Update CODEOWNERS
josh-chamberlain May 17, 2024
ad5b51b
Refactor DataRequestsManager, add DBRequestMapper class and tests
maxachis May 17, 2024
3c21385
Change psycopg2 to psycopg2-binary
maxachis May 17, 2024
83835da
Temporarily comment out github action to prevent execution
maxachis May 17, 2024
5acc1ff
Uncomment test_api.yml
maxachis May 19, 2024
a40f55b
Comment test_api.yml
maxachis May 19, 2024
31f6400
Merge pull request #1 from Police-Data-Accessibility-Project/mc_testi…
maxachis May 19, 2024
cd4cd74
Merge pull request #5 from Police-Data-Accessibility-Project/mc_handl…
josh-chamberlain May 21, 2024
15e7a95
Merge remote-tracking branch 'Data-Sources-App-V2/main' into mc_issue…
maxachis May 22, 2024
2e19d92
Merge remote-tracking branch 'Data-Sources-App-V2/dev' into mc_issue_…
maxachis May 22, 2024
aed0d58
Post-merge clean-up
maxachis May 22, 2024
fe883a2
Merge remote-tracking branch 'Data-Sources-App-V2/main' into dev
maxachis May 22, 2024
d407ada
Reformat with Black
maxachis May 22, 2024
1057284
Merge branch 'dev' into mc_issue_281_middleware_god_function_actual
maxachis May 22, 2024
073f60d
Remove broken import
maxachis May 22, 2024
f3bb63d
Update data source queries in SearchTokens
maxachis May 22, 2024
82274c9
Merge pull request #9 from Police-Data-Accessibility-Project/mc_issue…
maxachis May 22, 2024
409d4a8
Add conftest.py for pytest fixtures
maxachis May 22, 2024
ebd3ef4
Add User model to middleware
maxachis May 22, 2024
ba4e4a8
Reformat with black
maxachis May 22, 2024
c092c5d
Add Flask and Flask-SQLAlchemy to requirements.txt
maxachis May 22, 2024
2f30d89
Correct requirements.txt
maxachis May 22, 2024
85e6fce
Correct requirements.txt
maxachis May 22, 2024
cbe2713
Merge pull request #12 from Police-Data-Accessibility-Project/mc_sqla…
maxachis May 22, 2024
58431f3
Refactor error handling to use custom exceptions
maxachis May 24, 2024
0f3cb7a
Add tests for custom exceptions in authentication process
maxachis May 24, 2024
11cc622
Reformat with Black
maxachis May 24, 2024
bff0e22
Merge pull request #7 from Police-Data-Accessibility-Project/mc_issue…
josh-chamberlain May 24, 2024
a8b88e0
Refactor create_app function in app.py
maxachis May 25, 2024
ddc9d26
Remove unused methods from PsycopgResource class.
maxachis May 25, 2024
80b55e3
Add thorough tests for all application endpoints
maxachis May 25, 2024
06db561
Refactor test_app_with_mock fixture in app_test.py
maxachis May 25, 2024
2b2f3a5
Replace tuple return type with namedtuple in security middleware
maxachis May 25, 2024
f53d8f8
Add Bandit Security Linting to GitHub Actions
maxachis May 26, 2024
fae254f
Update Bandit execution command in workflow
maxachis May 26, 2024
006cec2
Removed cat command from Bandit run in GitHub Actions
maxachis May 26, 2024
b931fc1
Adjustment
maxachis May 26, 2024
12a9ba9
Adjustment
maxachis May 26, 2024
e05fbc1
Update Bandit Security Linting trigger event
maxachis May 26, 2024
2f3a61d
Expand targets for Bandit Security Linting
maxachis May 26, 2024
07318bb
Merge pull request #17 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 26, 2024
f9e4852
Refactor SQL queries for enhanced security
maxachis May 26, 2024
299975d
Merge pull request #18 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 26, 2024
2672d5e
Merge pull request #16 from Police-Data-Accessibility-Project/mc_add_…
maxachis May 26, 2024
bd86884
Merge pull request #14 from Police-Data-Accessibility-Project/mc_add_…
maxachis May 27, 2024
daf1608
Remove test parameters from query functions
maxachis May 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
/client/ @joshuagraber
* @mbodeantor
* @josh-chamberlain
32 changes: 32 additions & 0 deletions .github/workflows/bandit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Bandit Security Linting

on: [pull_request]

jobs:
bandit:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install bandit

- name: Run Bandit
run: |
bandit -r middleware resources app.py

- name: Upload Bandit results
uses: actions/upload-artifact@v2
with:
name: bandit-report
path: bandit_output.txt

2 changes: 1 addition & 1 deletion .github/workflows/pull.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Test with pytest
run: |
pip install pytest pytest-cov
pytest app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
pytest tests/resources/app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html

setup_client:
defaults:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/python_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ jobs:
uses: reviewdog/action-flake8@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
flake8_args: --ignore E501,W291 # Does not check for max line exceed or trailing whitespace
level: warning
25 changes: 25 additions & 0 deletions .github/workflows/test_api.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#name: Test API using Pytest
#
#on:
# pull_request:
#
#jobs:
# test_api:
# env:
# SECRET_KEY: ${{ secrets.SECRET_KEY }}
# DEV_DB_CONN_STRING: ${{secrets.DEV_DB_CONN_STRING}}
# name: Test API
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v4
# - uses: actions/setup-python@v4
# with:
# python-version: '3.11'
# - name: Install dependencies
# run: |
# python -m pip install --upgrade pip
# pip install -r requirements.txt
# python -m spacy download en_core_web_sm
# pip install pytest pytest-cov
# - name: Run tests
# run: pytest tests --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# data-sources-app
# data-sources-app-v2

Development of the next big iteration of the data sources app according to https://github.com/Police-Data-Accessibility-Project/data-sources-app/issues/248

An API and UI for searching, using, and maintaining Data Sources.

Expand Down Expand Up @@ -102,9 +104,18 @@ npm run dev

## Testing

All unit tests for the API live in the app_test.py file. It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated. Tests are currently run with pytest and can be run locally with the `pytest` command.
### Location
All unit and integration tests for the API live in the `tests` folder

It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated.


### How to run tests
Some tests involve interfacing with the development database, which copies the production database's data and schema daily.

To ensure such tests properly connect to the database, create or amend an `.env` file in the root direct of the project with the environment variable `DEV_DB_CONN_STRING`. Provide as a value a connection string giving you access to the `data_sources_app` user. If you do not have this connection string, DM a database administrator.

Endpoints are structured for simplified testing and debugging. Code for interacting with the database is contained in a function suffixed with "_results" and tested against a local sqlite database instance. Limited rows (stored in the DATA_SOURCES_ROWS and AGENCIES_ROWS variables in app_test_data.py) are inserted into this local instance on setup, you may need to add additional rows to test other functionality fully.
Tests are currently run with pytest and can be run locally with the `pytest` command.

Remaining API code is stored in functions suffixed with "_query" tested against static query results stored in app_test_data.py. Tests for hitting the endpoint directly should be included in regular_api_checks.py, makes sure to add the test function name in the list at the bottom so it is included in the Github actions run every 15 minutes.

Expand Down
6 changes: 2 additions & 4 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@
api.add_resource(resource, endpoint, resource_class_kwargs=kwargs)


def create_app() -> Flask:
psycopg2_connection = initialize_psycopg2_connection()

def create_app(psycopg2_connection) -> Flask:

Check warning on line 28 in app.py

View workflow job for this annotation

GitHub Actions / Lint

[flake8] reported by reviewdog 🐶 Missing docstring in public function Raw Output: ./app.py:28:1: D103 Missing docstring in public function
app = Flask(__name__)
api = Api(app)
CORS(app)
Expand Down Expand Up @@ -57,5 +55,5 @@


if __name__ == "__main__":
app = create_app()
app = create_app(initialize_psycopg2_connection())
app.run(debug=True, host="0.0.0.0")
Loading
Loading