Merge branch 'master' into production-server-e2e

docs/README.md

@@ -34,3 +34,4 @@ Here are some documents important for TEAMMATES developers.
 * [Setting up third-party email providers](emails.md).
 * [Setting up CAPTCHA](captcha.md).
 * [Snapshot testing](snapshot-testing.md)
+* [E2E testing](e2e-testing.md)

docs/development.md

@@ -159,48 +159,9 @@ POST http://localhost:8080/_ah/login?action=Log+Out
 
 There are two big categories of testing in TEAMMATES:
 - **Component tests**: white-box unit and integration tests, i.e. they test the application components with full knowledge of the components' internal workings. This is configured in `src/test/resources/testng-component.xml` (back-end) and `src/web/jest.config.js` (front-end).
-- **E2E (end-to-end) tests**: black-box tests, i.e. they test the application as a whole without knowing any internal working. This is configured in `src/e2e/resources/testng-e2e.xml`.
+- **E2E (end-to-end) tests**: black-box tests, i.e. they test the application as a whole without knowing any internal working. This is configured in `src/e2e/resources/testng-e2e.xml`. To learn more about E2E tests, refer to this [document](https://github.com/TEAMMATES/teammates/blob/master/docs/e2e-testing.md).  
 
-### Configuring browsers for E2E Testing
-
-TEAMMATES E2E testing requires Firefox or Chrome.
-
-Before running tests, modify `src/e2e/resources/test.properties` if necessary, e.g. to configure which browser and test accounts to use.
-
-#### Using Firefox
-
-* You need to use geckodriver for testing with Firefox.
-  * Download the latest stable geckodriver from [here](https://github.com/mozilla/geckodriver/releases).
-    The site will also inform the versions of Firefox that can be used with the driver.
-  * Specify the path to the geckodriver executable in `test.geckodriver.path` value in `test.properties`.
-
-* If you want to use a Firefox version other than your computer's default, specify the custom path in `test.firefox.path` value in `test.properties`.
-
-* If you are planning to test changes to JavaScript code, disable JavaScript caching for Firefox:
-  * Enter `about:config` into the Firefox address bar and set `network.http.use-cache` (or `browser.cache.disk.enable` in newer versions of Firefox) to `false`.
-
-#### Using Chrome
-
-* You need to use chromedriver for testing with Chrome.
-  * Download the latest stable chromedriver from [here](https://sites.google.com/a/chromium.org/chromedriver/downloads).
-    The site will also inform the versions of Chrome that can be used with the driver.
-  * Specify the path to the chromedriver executable in `test.chromedriver.path` value in `test.properties`.
-
-* If you are planning to test changes to JavaScript code, disable JavaScript caching for Chrome:
-  * Press Ctrl+Shift+J to bring up the Web Console.
-  * Click on the settings button at the bottom right corner.
-  * Under the General tab, check "Disable Cache".
-
-* The chromedriver process started by the test suite will not automatically get killed after the tests have finished executing.<br>
-  You will need to manually kill these processes after the tests are done.
-  * On Windows, use the Task Manager or `taskkill /f /im chromedriver.exe` command.
-  * On OS X, use the Activity Monitor or `sudo killall chromedriver` command.
-
-### Running the tests
-
-- When running the test cases, a few cases may fail (this can happen due to timing issues). They can be re-run until they pass without affecting the accuracy of the tests.
-
-#### Running the tests with command line
+#### Running the tests
 
 To run all front-end component tests in watch mode (i.e. any change to source code will automatically reload the tests), run the following command:
 ```sh
@@ -216,42 +177,19 @@ To run an individual test in a test file, change `it` in the `*.spec.ts` file to
 
 To run all tests in a test file (or all test files matching a pattern), you can use Jest's watch mode and filter by filename pattern.
 
-Back-end component tests and E2E tests follow this configuration:
+Back-end component tests follow this configuration:
 
 Test suite | Command | Results can be viewed in
 ---|---|---
 `Component tests` | `./gradlew componentTests` | `{project folder}/build/reports/tests/componentTests/index.html`
-`E2E tests` | `./gradlew e2eTests` | `{project folder}/build/reports/e2e-test-try-{n}/index.html`, where `{n}` is the sequence number of the test run
 Any individual component test | `./gradlew componentTests --tests TestClassName` | `{project folder}/build/reports/tests/componentTests/index.html`
-Any individual E2E test | `./gradlew e2eTestTry1 --tests TestClassName` | `{project folder}/build/reports/e2e-test-try-1/index.html`
-
-- `E2E tests` will be run in their entirety once and the failed tests will be re-run a few times. All other test suites will be run once and only once.
-- Before running `E2E tests`, it is important to have the both front-end and back-end dev servers running locally first if you are testing against them.
 
 You can generate the coverage data with `jacocoReport` task after running tests, e.g.:
 ```sh
 ./gradlew componentTests jacocoReport
 ```
 The report can be found in the `build/reports/jacoco/jacocoReport/` directory.
 
-### Testing against production server
-
-If you are testing against a production server (staging server or live server), some additional tasks need to be done.
-
-1. You need to setup a `Gmail API`<sup>1</sup> as follows:
-   * [Obtain a Gmail API credentials](https://github.com/TEAMMATES/teammates-ops/blob/master/platform-guide.md) and download it.
-   * Copy the file to `src/e2e/resources/gmail-api` (create the `gmail-api` folder) of your project and rename it to `client_secret.json`.
-   * It is also possible to use the Gmail API credentials from any other Google Cloud Platform project for this purpose.
-
-1. Edit `src/e2e/resources/test.properties` as instructed is in its comments.
-   * In particular, you will need legitimate Google accounts to be used for testing.
-
-1. Run the full test suite or any subset of it as how you would have done it in dev server.
-   * Do note that the GAE daily quota is usually not enough to run the full test suite, in particular for accounts with no billing enabled.
-
-<sup>1</sup> This setup is necessary because our test suite uses the Gmail API to access Gmail accounts used for testing (these accounts are specified in `test.properties`) to confirm that those accounts receive the expected emails from TEAMMATES.
-This is needed only when testing against a production server because no actual emails are sent by the dev server and therefore delivery of emails is not tested when testing against the dev server.
-
 ## Deploying to a staging server
 
 > `Staging server` is the server instance you set up on Google App Engine for hosting the app for testing purposes.

docs/e2e-testing.md

@@ -0,0 +1,141 @@
+# E2E Testing  
+
+* [What is E2E Testing](#what-is-e2e-testing)
+* [Running E2E Tests](#running-e2e-tests)
+    * [Configuring browsers for E2E Testing](#configuring-browsers-for-e2e-testing)
+    * [Running the tests](#running-the-tests)
+    * [Testing against production server](#testing-against-production-server)
+* [Creating E2E Tests](#creating-e2e-tests)
+    * [Page Object Pattern](#page-object-pattern)
+    * [Creating Page Objects](#creating-page-objects)
+    * [Things to avoid when writing E2E tests](#things-to-avoid-when-writing-e2e-tests)
+    * [FAQ](#faq)
+
+## What is E2E Testing?
+
+E2E (End-to-end) testing is a testing methodology where the objective is to test the application as a whole.   
+- It aims to ensure all integrated components of the application work together as expected when it is being used by the end user.   
+- This is done by simulating user scenarios on the fully built product.  
+
+E2E tests in TEAMMATES can be found in the package `teammates.e2e`. 
+
+## Running E2E tests
+
+### Configuring browsers for E2E Testing
+
+TEAMMATES E2E testing requires Firefox or Chrome.
+
+Before running tests, modify `src/e2e/resources/test.properties` if necessary, e.g. to configure which browser and test accounts to use.
+
+#### Using Firefox
+
+* You need to use geckodriver for testing with Firefox.
+  * Download the latest stable geckodriver from [here](https://github.com/mozilla/geckodriver/releases).
+    The site will also inform the versions of Firefox that can be used with the driver.
+  * Specify the path to the geckodriver executable in `test.geckodriver.path` value in `test.properties`.
+
+* If you want to use a Firefox version other than your computer's default, specify the custom path in `test.firefox.path` value in `test.properties`.
+
+* If you are planning to test against a production server, specify the Firefox profile to be used in `test.firefox.profile.name` value in `test.properties`.
+  * This is used to bypass login by using previous login data.
+  * You can enter `about:profiles` into Firefox address bar to identify the profile being used.
+
+#### Using Chrome
+
+* You need to use chromedriver for testing with Chrome.
+  * Download the latest stable chromedriver from [here](https://sites.google.com/a/chromium.org/chromedriver/downloads).
+    The site will also inform the versions of Chrome that can be used with the driver.
+  * Specify the path to the chromedriver executable in `test.chromedriver.path` value in `test.properties`.
+
+* If you are planning to test against a production server, specify the path to Chrome's user data directory in `test.chrome.userdata.path` value in `test.properties`.
+  * This is used to bypass login by using previous login data.
+
+* The chromedriver process started by the test suite will not automatically get killed after the tests have finished executing.<br>
+  You will need to manually kill these processes after the tests are done.
+  * On Windows, use the Task Manager or `taskkill /f /im chromedriver.exe` command.
+  * On OS X, use the Activity Monitor or `sudo killall chromedriver` command.
+
+### Running the tests
+E2E tests follow this configuration:
+
+Test suite | Command | Results can be viewed in
+---|---|---
+`E2E tests` | `./gradlew e2eTests` | `{project folder}/build/reports/e2e-test-try-{n}/index.html`, where `{n}` is the sequence number of the test run
+Any individual E2E test | `./gradlew e2eTestTry1 --tests TestClassName` | `{project folder}/build/reports/e2e-test-try-1/index.html`
+
+- `E2E tests` will be run in their entirety once and the failed tests will be re-run a few times. 
+- Before running `E2E tests`, it is important to have the dev server running locally first if you are testing against it.
+- When running the test cases, a few cases may fail (this can happen due to timing issues). They can be re-run until they pass without affecting the accuracy of the tests.
+
+### Testing against production server
+
+If you are testing against a production server (staging server or live server), some additional tasks need to be done.
+
+1. You need to setup a `Gmail API`<sup>1</sup> as follows:
+   * [Obtain a Gmail API credentials](https://github.com/TEAMMATES/teammates-ops/blob/master/platform-guide.md) and download it.
+   * Copy the file to `src/e2e/resources/gmail-api` (create the `gmail-api` folder) of your project and rename it to `client_secret.json`.
+   * It is also possible to use the Gmail API credentials from any other Google Cloud Platform project for this purpose.
+
+1. Edit `src/e2e/resources/test.properties` as instructed is in its comments.
+   * In particular, you will need a legitimate Gmail account to be used for testing.
+
+1. Login manually to TEAMMATES on the browser used for testing to add cookie with login details to the browser profile.
+   * This profile will be added to the web driver so that E2E tests will start with user already logged in. 
+   * This is required as Google does not allow login by automated software.
+
+1. For Firefox, run the full test suite or any subset of it as how you would have done it in dev server. 
+   * Do note that the GAE daily quota is usually not enough to run the full test suite, in particular for accounts with no billing enabled.
+
+1. For Chrome, you may have to run tests one at a time as multiple ChromeDriver instances cannot be opened with the same user data.
+
+<sup>1</sup> This setup is necessary because our test suite uses the Gmail API to access the Gmail account used for testing (the account is specified in `test.properties`) to confirm that the account receives the expected emails from TEAMMATES.
+This is needed only when testing against a production server because no actual emails are sent by the dev server and therefore delivery of emails is not tested when testing against the dev server.
+
+## Creating E2E tests
+
+As E2E tests should be written from the end user perspective, each test case should reflect some user workflow.   
+
+In TEAMMATES, E2E test cases are organized by page. For each page, we:  
+1. Identify the important user workflows  
+1. Simulate the user actions involved in the workflow by interacting with the UI elements.  
+1. Assert the expected conditions are present after the interaction.  
+
+[Selenium](https://www.selenium.dev/) is used to locate and interact with elements in the UI.   
+
+All E2E test classes inherit from `BaseE2ETestCase` which contains methods that are common to most test cases, such as preparing the `Browser` object used for testing.   
+
+To help verify the state of the datastore, `BackDoor` contains methods to create API calls to the back-end without going through the UI. 
+
+### Page Object Pattern
+
+In order to make E2E testing more robust to UI changes, the [Page Object Pattern](https://martinfowler.com/bliki/PageObject.html) is adopted.  
+
+Each page in TEAMMATES is represented by a page object class. The page object class abstracts interactions with UI elements and only exposes the functionality of each page as methods.   
+- This way only the page object classes require updating when there are UI changes  
+- Without Page Object Pattern, all test cases that use the changed UI element would require updating  
+
+To maximise the effectiveness of Page Object Pattern, interaction with UI elements should not occur outside the page objects.   
+
+
+### Creating Page Objects
+
+The page object should have methods to represent the main functionality of the page that testers can use to simulate user actions.   
+- The public methods for page objects should avoid exposing the UI elements it interacts with and instead focus on the functionality of the webpage.   
+- For example, instead of having methods like `fillSearchBox` and `clickSearchButton`, it is better to have a method `searchForInstructor` which hides the UI elements used.    
+
+All Page Object classes inherit from `AppPage` which contains methods that are common for interacting with the web elements such as filling in textboxes.   
+
+
+### Things to avoid when writing E2E tests
+
+1. **Testing based on implementation** - The focus should be on user actions instead of implementation details. Therefore, black box testing should be adopted and test cases should be designed around use cases.   
+1. **Excessive exception testing** - Testing edge cases with E2E tests should be avoided. This is because E2E tests are expensive to run and not that effective for isolating bugs. Hence we should focus on the happy path and exception paths that are more common. We should leave more exhaustive testing to lower-level unit or integration tests.   
+1. **Not following “Tell Don’t Ask" Principle** - Instead of “asking” for data from the page objects and performing operations on them, “tell” the page object to do the operations. This is mostly seen in the verification methods where assertions are done in the page object instead of in the test case. This improves readability and maintainability as data and behavior are placed together.  
+
+### FAQ  
+
+**Why are all the tests done in one `testAll()` method?**  
+We bundle together everything as one test case instead of having multiple test cases. The advantage is that the time for the whole test class will be reduced because we minimize repetitive per-method setup/teardown. The downside is that it increases the time spent on re-running failed tests as the whole class has to be re-run. We opt for this approach because we expect tests to pass more frequently than to fail.  
+
+**Why is there one JSON file for each test case?**  
+Each test case has its own JSON file and the data inside has a unique prefix to prevent clashes in the database that may cause test failure, since tests are run concurrently.

src/e2e/java/teammates/e2e/cases/AdminAccountsPageE2ETest.java

@@ -21,6 +21,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
 
         String googleId = "tm.e2e.AAccounts.instr2";

src/e2e/java/teammates/e2e/cases/AdminHomePageE2ETest.java

@@ -18,6 +18,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         AppUrl url = createUrl(Const.WebPageURIs.ADMIN_HOME_PAGE);
         AdminHomePage homePage = loginAdminToPage(url, AdminHomePage.class);

src/e2e/java/teammates/e2e/cases/AdminSearchPageE2ETest.java

@@ -29,7 +29,8 @@ protected void prepareTestData() {
     }
 
     @Test
-    public void allTests() {
+    @Override
+    public void testAll() {
         AppUrl url = createUrl(Const.WebPageURIs.ADMIN_SEARCH_PAGE);
         searchPage = loginAdminToPage(url, AdminSearchPage.class);
 

src/e2e/java/teammates/e2e/cases/AdminSessionsPageE2ETest.java

@@ -67,6 +67,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
 
         ______TS("verify loaded data");

src/e2e/java/teammates/e2e/cases/AutomatedSessionRemindersE2ETest.java

@@ -56,27 +56,31 @@ protected void prepareTestData() {
     // In all the tests, we need to explicitly log in as admin
     // because the cron job URLs are additionally protected by admin constraint in web.xml
     // and adding backdoor key is not sufficient to bypass it
-
     @Test
-    public void testFeedbackSessionOpeningReminders() {
+    @Override
+    public void testAll() {
+        testFeedbackSessionOpeningReminders();
+        testFeedbackSessionClosingReminders();
+        testFeedbackSessionClosedReminders();
+        testFeedbackSessionPublishedReminders();
+    }
+
+    private void testFeedbackSessionOpeningReminders() {
         AppUrl openingRemindersUrl = createUrl(Const.CronJobURIs.AUTOMATED_FEEDBACK_OPENING_REMINDERS);
         loginAdminToPage(openingRemindersUrl, GenericAppPage.class);
     }
 
-    @Test
-    public void testFeedbackSessionClosingReminders() {
+    private void testFeedbackSessionClosingReminders() {
         AppUrl closingRemindersUrl = createUrl(Const.CronJobURIs.AUTOMATED_FEEDBACK_CLOSING_REMINDERS);
         loginAdminToPage(closingRemindersUrl, GenericAppPage.class);
     }
 
-    @Test
-    public void testFeedbackSessionClosedReminders() {
+    private void testFeedbackSessionClosedReminders() {
         AppUrl closedRemindersUrl = createUrl(Const.CronJobURIs.AUTOMATED_FEEDBACK_CLOSED_REMINDERS);
         loginAdminToPage(closedRemindersUrl, GenericAppPage.class);
     }
 
-    @Test
-    public void testFeedbackSessionPublishedReminders() {
+    private void testFeedbackSessionPublishedReminders() {
         AppUrl publishedRemindersUrl = createUrl(Const.CronJobURIs.AUTOMATED_FEEDBACK_PUBLISHED_REMINDERS);
         loginAdminToPage(publishedRemindersUrl, GenericAppPage.class);
     }

src/e2e/java/teammates/e2e/cases/BaseE2ETestCase.java

@@ -60,6 +60,8 @@ protected void prepareBrowser() {
 
     protected abstract void prepareTestData() throws Exception;
 
+    protected abstract void testAll();
+
     @Override
     protected String getTestDataFolder() {
         return TestProperties.TEST_DATA_FOLDER;

src/e2e/java/teammates/e2e/cases/FeedbackConstSumOptionQuestionE2ETest.java

@@ -30,6 +30,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         testEditPage();
         testSubmitPage();

src/e2e/java/teammates/e2e/cases/FeedbackConstSumRecipientQuestionE2ETest.java

@@ -31,6 +31,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         testEditPage();
         testSubmitPage();

src/e2e/java/teammates/e2e/cases/FeedbackContributionQuestionE2ETest.java

@@ -32,6 +32,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         testEditPage();
         testSubmitPage();

src/e2e/java/teammates/e2e/cases/FeedbackMcqQuestionE2ETest.java

@@ -31,6 +31,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         testEditPage();
         testSubmitPage();

src/e2e/java/teammates/e2e/cases/FeedbackMsqQuestionE2ETest.java

@@ -33,6 +33,7 @@ protected void prepareTestData() {
     }
 
     @Test
+    @Override
     public void testAll() {
         testEditPage();
         testSubmitPage();