Replace test query results #302
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
An exception handling decorator named handle_exceptions was introduced to simplify the handling of exceptions in most major resources. The decorator eliminates the need to manually try/except blocks in each route, thus reducing repetitive code and enhancing readability. All routes in resources like User.py, ResetTokenValidation.py, RequestResetPassword.py, and more were updated to use this decorator.
This commit adds a new python file containing unit tests for middleware functions that interact with the database. These tests include checking for the addition, deletion, retrieval, and update of various data sources, users, and reset tokens to ensure the correct operation and accuracy of these functions.
This commit introduces a new helper file for testing middleware interaction with the database. It includes test setup methods for generating fake data in the database, such as users, data sources, and reset tokens, which will provide a solid basis for thorough testing of database-related middleware functionality.
This commit both minimizes test implementation detail within individual tests and also enhances testing utilities. Specifically, large helper functions for creating test users and reset tokens have been moved from the test file into a separate helper file. Additionally, a new comprehensive test for the 'quick_search_query' function has been introduced to ensure proper functionality.
Renamed 'helper_test_middleware.py' to 'middleware/helper_functions.py', and made modifications to test data inserts. Added new testing helper functions for quick search query logs and boolean dictionary creation. This commit includes renaming a test file along with additions and modifications to the test data insert statements. New test helper functions have been introduced to handle the quick search query logs and create a dictionary of booleans. This refactor aims at improving the test coverage and overall project organization.
Added two testing fixtures for middleware tests in a new `fixtures.py` file within tests/middleware directory. The fixtures provide functionality for setting up a connection to the development database and creating a cursor for database operations. They also ensure that changes made during tests can be rolled back using PostgreSQL's SAVEPOINT management. This is done to prevent any permanent changes to the database resulting from the testing process.
Added several new tests for data source query functions in a new file, `test_data_source_queries.py`. These tests cover different aspects like getting approved data sources, data source by id and data sources for map. Additional setup and teardown functionality were also added through two fixtures, 'inserted_data_sources_found' and 'connection_with_test_data' for necessary test setup and cleanup processes.
Moved the 'connection_with_test_data' fixture from 'test_data_source_queries.py' to 'fixtures.py' to improve organization and reusability..
This commit introduces two new tests for the archives_get_results and archives_get_columns methods in the middleware layer. These tests validate the method returns and interaction with the database, enhancing the overall test coverage.
This commit introduces tests for the login_queries middleware, specifically for the login_results, create_session_token, and is_admin methods. These new tests ensure these methods interact properly with the database and improve the test coverage in our middleware layer.
Added optional return type and function description for get_most_recent_quick_search_query_log in tests/middleware/helper_functions.py. Also, handling no-result situation by checking if the result is None. These enhancements improve function's readability and robustness.
Implemented unit tests for the quick search query methods, specifically `unaltered_search_query` and `quick_search_query`. These tests ensure that search query results and logging function properly and return expected outputs. This includes checks on the length of responses, expected keys, and datetime accuracy in logs.
Implemented unit tests for user and reset token query methods in the middleware. These tests validate the functionality of creating, checking, and deleting users and reset tokens, ensuring the correct interaction with the database. The tests also validate the return values and the changes made to the database state.
The README has been updated to reflect changes in the project's testing structure. The new instructions include details on where unit and integration tests for the API are located, and the need to set up an environment variable for tests requiring database access. The instructions on how to run the tests, using the `pytest` command, and their interaction with CI/CD pipelines, remain unchanged.
The Python test workflow file has been renamed from python_tests.yml to test_api.yml and fully configured to run tests using GitHub Actions. It's been set up to specifically test the project's API on pull requests with environment variables for secrets and the pytest command to run the tests. It also upgrades pip and downloads dependencies.
The test imports in test_data_source_queries.py have been updated. This revision includes modifying the import path for app_test_data and removing a redundant import from middleware.fixtures.
The test files have been relocated to tests/resources directory and their respective import paths have been updated accordingly. Additionally, unused imports have been removed from the middleware tests, thus enhancing code readability and maintenance.
Optimized code for better readability through improving docstrings and line break formatting. Added flake8 argument to ignore E501 (line too long) to avoid warnings related to line exceeding maximum length.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
A custom exception, DatabaseInitializationError, has been added for scenarios where psycopg2 connection initialization fails. The exception handling in the `initialize_psycopg2_connection` function and usage in other places has been adapted accordingly. Furthermore, skeleton tests for successful and unsuccessful psycopg2 connection initialization have been added.
Updated the middleware to handle the UserNotFoundError and implemented the custom exception for when a user is not found in the login_queries script. Also refined the return type of is_admin function to only boolean, managing the missing user situation with the new custom exception.
The code has been refactored to individual query methods for clear data handling and improve readability. The 'data_sources_query' method in 'DataSources.py' which was previously handling all query types is replaced with specific, self-explanatory query methods such as 'get_approved_data_sources', 'needs_identification_data_sources' and 'get_data_sources_for_map'. This also leads to the introduction of 'convert_data_source_matches' in 'data_source_queries.py' file for better format conversion.
The archives_put_query method has been removed from the archives_queries.py and its functionality has been split into direct calls to the appropriate methods, archives_put_broken_as_of_results or archives_put_last_cached_results. This change simplifies the code and makes it more legible by clearly specifying the function being used in each situation. The refactoring also involves adjustments in the Archives.py resource to accommodate these changes.
In the helper function and tests module, the SQL queries were updated to use `DATETIME_OF_REQUEST` instead of `UPDATED_AT`. In `test_quick_search_query.py`, the code to get the current date was changed to pull directly from the database using 'SELECT NOW()'. Removed redundancy in data_source_by_id_query assertions and simplified datetime comparison in quick_search_query test. Made necessary import adjustments in `test_data_source_queries.py`.
…lchemy Mc sqlalchemy
Several functions that previously returned errors as dictionary keys have been refactored to instead raise custom exceptions. A new file 'custom_exceptions.py' has been created to define these exceptions, which are UserNotFoundError and TokenNotFoundError. This change enhances readability, and makes error handling more effective and explicit in the codebase.
Added new test cases to validate and ensure the proper functioning of the recently implemented custom exceptions in the middleware. These tests ensure that UserNotFoundError and TokenNotFoundError are correctly raised during the authentication process when the user or token does not exist, respectively.
…_276_refactor_middleware_throw_exceptions Added custom exception for psycopg2 connection initialization
The refactoring involved moving the initialization of the psycopg2 connection out of the create_app function and into its parameter list. This allows the function to be more flexible and reusable, capable of accepting different database connections. It allows for easier testing and potential use with different database management systems in the future.
This commit includes the creation of a new file, test_endpoints.py, specifically dedicated to thoroughly testing the functionality of all application endpoints. It utilizes Pytest to ensure that each endpoint correctly calls (or doesn’t call) the appropriate methods in their supporting classes, as per the original design. The test checks both allowed and not allowed methods for each endpoint.
The test_app_with_mock fixture in app_test.py has been simplified. The mocker from pytest-mock has been utilized to replace the previously used patch and MagicMock. Node assertions for database interaction have been removed from the endpoint test function.
The function 'is_valid' in the security middleware has been refactored. Instead of returning a Tuple, it now returns an instance of APIKeyStatus, which is a namedtuple. This namedtuple structure improves readability and understanding of the code by labeling the boolean values it returns.
A new GitHub Actions workflow 'bandit.yaml' has been added to run Bandit, a Python security linter, on push and pull requests. This will increase the security standards of the codebase by ensuring that potential security vulnerabilities are addressed promptly. The Bandit results will be uploaded as an artifact for further analysis.
Removed the explicit exit command from the Bandit execution command in the GitHub Actions workflow.
The command to print out Bandit's output directly to the console in the GitHub Actions workflow was removed. Now, the results are only being uploaded as an artifact for review.
Changed the triggering event of Bandit Security Linting workflow from happening on both push and pull_request events to only on pull_request events.
Updated the Bandit Security Linting code to also include 'resources' and 'app.py', expanding the range of files which are scanned for potential security vulnerabilities. Previously, linting was only applied to 'middleware'.
…e_292_bandit Add Bandit Security Linting to Github Actions
Modified SQL queries across multiple sources to use parameterized inputs instead of string formatting. This approach enhances security by preventing potential SQL injection attacks. The updates were applied to queries in files such as `login_queries.py`, `user_queries.py`, `reset_token_queries.py`, and others. Additionally, relevant tests were added to assure correct functioning.
…e_212_paramerize_sql_queries Refactor SQL queries for enhanced security
…named_tuples Replace tuple return type with namedtuple in security middleware
…endpoint_tests Mc add endpoint tests
The query methods in the middleware and resource files are updated to remove the test parameters, leading to simplified code. The code accessibility is improved by not allowing externally supplied test results to be processed, reducing the possibility of incorrect outputs. All functions now rely only on database queries for data, resulting in a more robust system.
|
Dangit, wrong fork again |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GitHub pull request template
Fixes
Description
Testing
Performance
Docs